Doing fullscreen colormaps in postprocessing?

[Graf Zahl]

This is something I have been thinking about since the addition of the postprocessing stuff:

At the moment, modifications like the invulnerability colormap are done during the actual render pass.
This causes some rather unpleasant visual artifacts, most importantly screwing up additively blended textures.

So what I would like to do is to move this into the postprocessing, where the effect can be applied to the entire scene so that these artifacts can go away.
The question now is, where's the best point to hook this in? It should only affect the 3D scene and the weapon HUD sprite.

[dpJudas]

In that case it should be done just after TonemapScene() or LensDistortScene() at: https://github.com/coelckers/gzdoom/blo ... e.cpp#L861. I've been wanting to move those 4 calls to a single PostProcessScene() function in gl_postprocess.cpp for a slightly better hand off between scene render and post process.

There is one small thing to look out for. In my (lack of?) wisdom, the effect buffers are as big as the entire screen. This means that when you're sampling from the scene texture you have to offset and scale accordingly. Same when setting up the viewport. The tonemap shader is actually not taking this into account right now and just tonemaps the entire thing. It only gets away with it because the status bar and 2D background is painted on top afterwards. You should be able to get away with the same for the colormaps.

[Nash]

Sorry for the possibly unrelated question but I've also been wondering:

dpJudas, you've stated possible plans to implement SSAO. If SSAO is a screen post processing effect, does that mean that the AO will be rendered on TOP of the 3D viewport? IMO this might look off with coloured fog and is actually a very common problem I've seen even in professional modern games... whenever coloured fog is mixed with SSAO, it sorta looks off because the AO shadows are being rendered full-black on top of the coloured fog... just wondering what are you thoughts on this and if you have any strategies to avoid this, assuming you'd still be interested in adding AO?

[dpJudas]

You probably should have created a new thread.

Yes, screen space ambient occlusion works by adding shadows on top of the scene. It therefore has certain limitations like not working with transparent stuff. Typically the SSAO pass adds its shadows after the solid parts of a scene has been rendered before switching to transparent scene elements. The SSAO pass can take the fog into account, but this requires that the scene outputs fog information for the SSAO shader in one of its gbuffers. So it can deal with it, but it comes at a performance price, which is probably why you don't see many games doing it.

I'm actually working on adding AO right now. It is taking a little bit longer because unlike the other passes I did not have the code already for doing this. The initial version won't be dealing with the fog as the only gbuffer in gzdoom right now is the lit scene. Adding such stuff might require some planning because gzdoom has be able to fall back gracefully to older hardware where generating a bunch of gbuffers would be too expensive.

[Graf Zahl]

More importantly, I'd prefer to get a few other things sorted out before adding ever more passes to the postprocessing. This is all nice and well but due to having only a limited amount of time at my hands to work on GZDoom, right now I have no good overview what the engine is even doing here!

[dpJudas]

Okay I'll delay any pull requests on more postprocessing stuff for the time being. It is mostly experimental on my behalf right now anyway - only brought it up because Nash explicitly asked.

It sounded like about a month ago that you'd soon like to do a new release of gzdoom, which would be yet another reason for me to not try dump new passes into the codebase.

[Graf Zahl]

Good. I think right now the most important thing is to stabilize what's there and make sure that all configurations and render paths work as intended.
The only thing I want to address before a release is the colormap handling because that'd actually be a major improvement.

[dpJudas]

For the colormap handling, my approach would be to use FGLRenderer::TonemapScene() as my template for a new function. Only thing that really has to change is which shader it uses and the uniforms passed in.

[Graf Zahl]

Just added.

While testing this I noticed that apparently it's not possible to toggle gl_renderbuffers during gameplay. If it is switched from 1 to 0, nothing gets drawn anymore. It looks like this isn't really switching off the external buffers completely and continues drawing into them, but not using the actual data afterward.

[dpJudas]

It is because of the way I check gl_renderbuffers. When you toggle it off IsEnabled immediately begins to return false, but this has the unfortunate side effect of it not binding back the default FBO if one of the other FBOs were bound at the time. I'll do a PR fixing this.

[Graf Zahl]

That's what CUSTOM_CVARs are there for, they can do some action on value change.

[dpJudas]

I'm not so keen on doing OpenGL calls in a CUSTOM_CVAR handler. It makes it a little bit unclear exactly when this code will fire. Already today the problem is that the gl_renderbuffers change happen in the middle of rendering. If doom had a more sane separation of concerns so that the game loop looked like while (!quit) { TickGame(); RenderGame(); ProcessInput(); } or similar the problem wouldn't have happened in the first place.

That said, we can fix it by just firing off "if (self == 0) glBindFramebuffer(GL_FRAMEBUFFER, 0)" in a CUSTOM_CVAR, if that's your preferred solution to it.

[dpJudas]

By the way, is the colormap thing fully hooked up yet? Just tried with the palette tonemap on and it looks like the flashes when picking up things are still done in the main shader.

[Graf Zahl]

In that case I suggest a two-stage approach: Don't ever check gl_renderbuffers directly but instead copy its value to another variable when a frame starts, do with the framebuffers what is necessary if it changes and use the copy for the internal checks.

[Graf Zahl]

By the way, is the colormap thing fully hooked up yet? Just tried with the palette tonemap on and it looks like the flashes when picking up things are still done in the main shader.

This is not what the colormap is doing. These are just normal blends that do not require any special treatment.
The change was for the invulnerability colormap which was suffering from some display anomalies due to how additive blending works.

But I moved the overlay that's used for color flashes to a later place in the renderer because it needs to occur after the colormap shader.