SSAO discussion

Advanced OpenGL source port fork from ZDoom, picking up where ZDoomGL left off.
[Home] [Download] [Git builds (Win)] [Git builds (Mac)] [Wiki] [Repo] [Bugs&Suggestions]

Moderator: Graf Zahl

Rachael
Developer
Developer
Posts: 3623
Joined: Sat May 13, 2006 10:30

SSAO discussion

Post by Rachael » Mon Aug 29, 2016 13:56

Starting this thread as a split from this one where Nash asked about SSAO.

Before we get too much into it, one problem we'll have with SSAO is that GZDoom is rendering sprites to the screen - which are meant to resemble solid objects. With standard SSAO filters that may not be too much of a problem but with the HBAO+ filter will actually use the geometry presented to the GPU.
Spoiler: Zen Sarcasm

User avatar
Nash
Developer
Developer
Posts: 1226
Joined: Sun Sep 25, 2005 1:49
Location: Kuala Lumpur, Malaysia
Contact:

Re: SSAO discussion

Post by Nash » Mon Aug 29, 2016 15:14

So does that mean sprites will end up having rectangular shadows instead of whatever pixel content is in the sprite? :S

Rachael
Developer
Developer
Posts: 3623
Joined: Sat May 13, 2006 10:30

Re: SSAO discussion

Post by Rachael » Mon Aug 29, 2016 15:16

Honestly, I am really not sure what sprites will end up having. I've never seen anyone do SSAO with sprites before.
Spoiler: Zen Sarcasm

User avatar
Nash
Developer
Developer
Posts: 1226
Joined: Sun Sep 25, 2005 1:49
Location: Kuala Lumpur, Malaysia
Contact:

Re: SSAO discussion

Post by Nash » Mon Aug 29, 2016 15:22

I haven't either, come to think of it. I've never seen billboarded sprites in games cast shadows and IMO that would look weird - imagine if things like explosions or particle FX casting AO Shadows..

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: SSAO discussion

Post by Graf Zahl » Mon Aug 29, 2016 15:42

Sprites are always rendered in the translucent pass, so I have no idea what this may imply. Also don't forget that the engine makes extensive use of stencils for portal rendering, which also leaves certain parts in a state that may produce incorrect results.

Rachael
Developer
Developer
Posts: 3623
Joined: Sat May 13, 2006 10:30

Re: SSAO discussion

Post by Rachael » Mon Aug 29, 2016 16:08

With SSAO itself (the super-sampling part of it, that is, that darkens corners), that may not be too huge of an issue, though there might be inappropriate shadow lines where there shouldn't be at the portal borders. As I said though, I don't think anyone is going to fret TOO much over it, especially considering SSAO will be an optional feature if it ever gets implemented.

With *actual* shadows though - like I think Nash is suggesting - that can present much bigger issues.
Spoiler: Zen Sarcasm

User avatar
Nash
Developer
Developer
Posts: 1226
Joined: Sun Sep 25, 2005 1:49
Location: Kuala Lumpur, Malaysia
Contact:

Re: SSAO discussion

Post by Nash » Mon Aug 29, 2016 16:34

I wasn't actually talking about casted shadows if that's what you thought I meant... I might have used the word "shadow" incorrectly in this thread but what I've been talking about all this while is indeed the darkening that happens when 2 surfaces are close to each other. :D

On another topic, I don't know what kind of ambient occlusion Doomsday Engine is using with their "FakeRadio" feature but I noticed that sprites aren't included with the FakeRadio... it's literally just the level that's being "shadowed".

Another interesting observation I noticed a while back when qeffects.dll was popular and trendy around here - enabling SSAO produces weird shadows in the sky/skybox. :O I had a skybox sector that is textured with a flat blue texture, and with Line_Horizon on the walls but I ended up seeing contact shadows in the sky...

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: SSAO discussion

Post by Graf Zahl » Mon Aug 29, 2016 16:59

What you saw with the skybox is precisely the stencil effects at play. To ensure that the actual level gets rendered properly sky drawing is doing the following:

Step 1: Draw the stencil
Step 2: Draw the sky into the stencil buffer (or as a separate pass into the main buffer if there's only one sky)
Step 3: Draw the stencil again, this time only affecting the depth buffer.

So you actually get 'edges' in the depth buffer where you see the sky.

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: SSAO discussion

Post by dpJudas » Mon Aug 29, 2016 21:53

GZDoom doesn't render the sprites into the depth buffer, so they will (luckily) not affect the darkening generated by a SSAO pass. The trickiest thing is handling the sky issue along with running the pass exactly between where it switches from solid to translucent objects. If the pass is run after the sprites have been rendered the darkening of things behind the sprites will overlay them.

If you're interested I can upload a build of the SSAO pass I have working in gzdoom right now. It probably looks pretty close to what you saw with qeffects at its current stage though. I'm not sure what is the best way to deal with the stencil/depth thing. Essentially there are two choices: either the main shader outputs linear depth values to a second color buffer (that would introduce the first real gbuffer in gzdoom). Or the SSAO shader looks at the stencil values and assumes there was no ray hit if the stencil changes between the two pixels examined. I'm not sure yet which of the two I prefer.

I don't know the details of HBAO+ right now (in 3D there's so many trending algorithms - can't keep up :D), but most of the algorithms purely use depth data. The SSAO algorithm I've implemented uses purely the depth buffer, but it does attempt to reconstruct a normal for the pixel by looking at 4 neighboring pixels. From there it does 32 ray tests around the pixel and dots that with the reconstructed normal. If that makes it HBAO or HBAO+ I'm not sure, but I'm pretty sure the original SSAO algorithm did not use any normal.

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: SSAO discussion

Post by Graf Zahl » Mon Aug 29, 2016 22:05

it's a bit more tricky than that because of portals in general. Unless something is done, everything behind a portal will show the same problems as the sky. Horizons will do so as well. Every time a portal is finalized the depth buffer will be rewritten in its place, and on top of that, sprites inside portals will be rendered before sprites in the main scene because a portal will be fully processed before going on to the next.

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: SSAO discussion

Post by dpJudas » Mon Aug 29, 2016 22:21

Okay that rules out the possibility to look at the stencil. It will have to output the real depth in a second color buffer for the SSAO pass to work. The sprites thing is certainly more tricky - but maybe it could write some info into the second color buffer so the SSAO pass will know this pixel is covered by a sprite and skip it. On the plus side that means the pass doesn't have to run in the middle of scene rendering.

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: SSAO discussion

Post by dpJudas » Tue Aug 30, 2016 0:25

Thinking about this a little further, I'm not sure flagging sprites in a color buffer will do the trick. It will probably work for monster sprites, but much more difficult with stuff like fireballs.

Maybe it could run a partial SSAO pass covering the portal entrance and then only work on the pixels marked by the stencil. A bit expensive maybe if there's a lot of overlapping portals, but would it work?

User avatar
Nash
Developer
Developer
Posts: 1226
Joined: Sun Sep 25, 2005 1:49
Location: Kuala Lumpur, Malaysia
Contact:

Re: SSAO discussion

Post by Nash » Tue Aug 30, 2016 14:47

dpJudas wrote: If you're interested I can upload a build of the SSAO pass I have working in gzdoom right now. It probably looks pretty close to what you saw with qeffects at its current stage though.
I'm interested, if only for curiosity purposes (and to see how much performance difference it is on my computer). :D

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: SSAO discussion

Post by dpJudas » Tue Aug 30, 2016 19:14

Okay, here it is: https://github.com/dpjudas/zdoom/releases/tag/ssao1

Note that the pass has some outstanding issues that causes it to create some annoying shadows due to some bug in the shader itself. You can see how the SSAO part looks like using ssao_debug 1. The gaussian blur also needs to be depth aware.

Overall though, without a solution to the portal situation, the entire thing is more or less useless.

User avatar
Nash
Developer
Developer
Posts: 1226
Joined: Sun Sep 25, 2005 1:49
Location: Kuala Lumpur, Malaysia
Contact:

Re: SSAO discussion

Post by Nash » Wed Aug 31, 2016 3:31

I know it's just a test build but while we're here, might as well...

http://imgur.com/a/DtDWw

The shadow position seems to not match the 3D scene. All in all, this was pretty cool to look at, even in the debug mode. I'm amazed at how it handles the transparent wall textures so well, for some reason. :mrgreen:

Locked

Return to “GZDoom”