Stereo 3D stuff

[Graf Zahl]

I was sorta wishing that dpJudas would also create an example of how to create and use a custom present shader, but perhaps I'll need to figure that out for myself.

It's not really that hard. You use two, not one texture as input and then mix them as you need:

Code: Select all

in vec2 TexCoord;
out vec4 FragColor;

uniform sampler2D InputTexture1;
uniform sampler2D InputTexture2;
uniform float InvGamma;
uniform float Contrast;
uniform float Brightness;
uniform vec4 leftColorFactor, rightColorFactor;

vec4 ApplyGamma(vec4 c)
{
	vec3 val = max(c.rgb * Contrast - (Contrast - 1.0) * 0.5, vec3(0.0));
	val = pow(val, vec3(InvGamma));
	val += Brightness * 0.5;
	return vec4(val, c.a);
}

void main()
{
	vec4 leftFragColor = ApplyGamma(texture(InputTexture1, TexCoord));
	vec4 rightFragColor = ApplyGamma(texture(InputTexture2, TexCoord));
	FragColor = leftFragColor * leftColorFactor + rightFragColor * rightColorFactor;
}

I think for this gamma should be done before combining the images. And of course, if you want you can do more complex processing of the two eye textures as well.

[dpJudas]

The one bug I found relates to quad buffered stereo. If I start gzdoom in quad stereo mode by including "+vr_mode 7" on the command line, the doom screen remains totally black, which is horrible. On the other hand, vr_mode 7 works correctly if I wait and activate it after beginning game play in gzdoom. It's possible this behavior is unchanged from previous versions.

Hmm, that is strange. Could you try add a glDrawBuffer(GL_BACK_LEFT) to the else section of QuadStereo::Present? (https://github.com/coelckers/gzdoom/blo ... eo.cpp#L62). If that fixes it (for the left eye), then the bQuadStereoSupported check in the constructor must be failing for some reason.

EDIT: Just noticed it does GLRenderer->mBuffers->BindEyeTexture(1, 0) in the else. That should be BindEyeTexture(0, 0) since the right eye is not available due to what the constructor is doing.

Even if the blending ends up not-quite-right, I believe this two-pass approach is best. I propose we wait and see what blending troubles occur, and then form a plan of attack. Hmm. This might explain some confusion I experienced with the cross-hair blending in GZ3Doom.

I'm not sure if adjusting 2DDrawer to render to a texture is easier. The reason is that the 2D drawer needs to have its blend mode output pre-multiplied alpha, because it can't do the actual final blend until the texture is rendered. Without that change it will darken transparent things instead of making them translucent.

On the other hand, drawing the 2D directly to the quad involves just changing the active matrix to this pseudo code:

transform = projectionMatrix * worldToEye * quadToWorld * transform(-0.5, 0.5) * scale(1/SCREENWIDTH, 1/SCREENHEIGHT, 1)

The only problem with this approach is the scissor clipping, but I think that can be moved to gl_ClipDistance in the shader without too much work.

I was surprised to notice that dpJudas did not need to create any new shaders to get this all working.

I was going to do exactly what Graf illustrates (with the two input textures) when I discovered I could just get away with the old colormask setup. Didn't see that coming until I got to that step.

[biospud]

The one bug I found relates to quad buffered stereo. If I start gzdoom in quad stereo mode by including "+vr_mode 7" on the command line, the doom screen remains totally black, which is horrible. On the other hand, vr_mode 7 works correctly if I wait and activate it after beginning game play in gzdoom. It's possible this behavior is unchanged from previous versions.

Hmm, that is strange. Could you try add a glDrawBuffer(GL_BACK_LEFT) to the else section of QuadStereo::Present? (https://github.com/coelckers/gzdoom/blo ... eo.cpp#L62). If that fixes it (for the left eye), then the bQuadStereoSupported check in the constructor must be failing for some reason.

EDIT: Just noticed it does GLRenderer->mBuffers->BindEyeTexture(1, 0) in the else. That should be BindEyeTexture(0, 0) since the right eye is not available due to what the constructor is doing.

Thanks for looking into this. OK, we can change the black-screen problem by changing the else clause as you suggest, to say:

Code: Select all

GLRenderer->mBuffers->BindEyeTexture(0, 0);

Now, entering "+vr_mode 7" on the command line results in a nice non-black, but mono, display. Presumably bQuadStereoSupported is being set to false in the constructor. Those glGet... calls need to be delayed until there is a complete valid OpenGL context.

[biospud]

I finally got this sorted with two changes:
* Fix to BindEyeTexture(0, 0) in quad-buffered mono branch, as suggested by dpJudas
* Delay choosing stereo vs. mono in quad-buffered 3d mode until a usable OpenGL context is first observed

Pull request here: https://github.com/coelckers/gzdoom/pull/97

[biospud]

Now that the stereo 3D modes for the upcoming release are pretty much nailed, I started a new branch for the side-by-side 3D modes. You can see what these are supposed to look like here (modes 3 and 4):
https://github.com/cmbruns/gz3doom/wiki ... #vr_mode-3

I am pleased that I was able to implement mode 4 (narrow side-by-side) flawlessly in just a few minutes, using this new renderbuffer infrastructure. I simply had to adjust the output viewport argument to GLRenderer->DrawPresentTexture(...).

But mode 3 (wide side-by-side) might be trickier. Because the composition of the various GUI elements needs to fit nicely into the very narrow aspect ratio of a half-width screen. This requires changing not just the output viewport, but also changing the framebuffer shape of the entire rendering pipeline, right up until just before presentation time. Any tips on how I might accomplish this?

[dpJudas]

The main function controlling the viewport used during rendering is FGLRenderer::SetOutputViewport (gl_renderer.cpp). It sets up three rectangles:

• mOutputLetterbox. This is the final glViewport that the present shader will use when copying to the back buffer.
• mScreenViewport. The glViewport used when rendering 2D stuff. The left/top values are always 0 here when render buffers are active.
• mSceneViewport. The glViewport used when rendering the scene itself.

The eye textures will be dimensioned after what size mScreenViewport has. So, in theory, all you have to do is calculate a different dimension in SetOutputViewport if you want a different size for a different vr_mode.

What complicates matters a little bit is the fact that there are a couple different ways a frame may be rendered:

1. There's the "we have no scene" mode which happens at launch. In this case what calls SetOutputViewport is either OpenGLFrameBuffer::Update, or OpenGLFrameBuffer::InitializeState for the first frame. In this case 'bounds' will be null. This mode also happens during wipes, I think.

2. The other way is when FGLRenderer::RenderViewpoint gets called. That is where the 'bounds' argument enters the picture because that function may be called by screenshot code, save game thumbnail, or normal scene render. Only in the normal scene version will bounds be null.

With all that explained, what you need is to setup different values for those three viewport variables in SetOutputViewport, but only do so if bounds is null. Otherwise you'll disrupt screenshots, thumbnails and potential render-to-texture stuff.

Hope this helps.

[biospud]

Thanks dpJudas for the explanation.
I think I can replace the SetOutputViewport() call in FGLRenderer::RenderViewpoint() with a 3D-mode-specific virtual function. I suspect that in the screenshot case, the conditional (mainview && toscreen) would be FALSE, so the stereo 3D mode would always be MONO, in which case the standard SetOutputViewport(bounds) would be invoked.

I'm going to defer the "we have no scene" case for the moment, and figure that one out later. But I know I'll need to go there eventually, before the HMD modes could work properly.
If I succeed in implementing this full-width side-by-side mode, then GZDOOM will have every 3D mode in my three-year-old-fork GZ3DOOM, except for one. At that point it will be time to move on to HMD modes.

My wife reminds me that we got SteamVR working with the Oculus Rift DK2 at one point. Though I remember the process involves ignoring some scary warnings and clicking "Next". So you might want to dust your DK2 off, in case my momentum here can be sustained.

[dpJudas]

I think I can replace the SetOutputViewport() call in FGLRenderer::RenderViewpoint() with a 3D-mode-specific virtual function. I suspect that in the screenshot case, the conditional (mainview && toscreen) would be FALSE, so the stereo 3D mode would always be MONO, in which case the standard SetOutputViewport(bounds) would be invoked.

Calling a virtual function like that should do nicely, but I'd highly recommend you do that call inside the SetOutputViewport function itself. Adding "s3d::Stereo3DMode::getCurrentMode()->AdjustViewports()" at the very end of FGLRenderer::SetOutputViewport should do the trick. That will also allow you to make the current modes do nothing in that function as they are already happy with the default calculations.

Because of the complex nature of viewport management in gzdoom, the mSceneViewport, mScreenViewport and mOutputLetterbox variables are really to be considered immutable by all other code than SetOutputViewport and child functions of it. In a future version I'll probably move it into its own class in a setup that prevents changing the variables. This is to prevent the situation from the old version where there was no coherent strategy to what piece of code is ultimately responsible for keeping that state correct and sane. For reasons of why this is so important, see the global variables in the software renderer.

My wife reminds me that we got SteamVR working with the Oculus Rift DK2 at one point. Though I remember the process involves ignoring some scary warnings and clicking "Next". So you might want to dust your DK2 off, in case my momentum here can be sustained.

It looked like on your wiki that I can still download drivers for the DK2 after CV1 was released, so I probably should undust it.

[biospud]

Adding "s3d::Stereo3DMode::getCurrentMode()->AdjustViewports()" at the very end of FGLRenderer::SetOutputViewport should do the trick

Thanks I'm adopting this approach but have not had much success yet.

It looked like on your wiki that I can still download drivers for the DK2 after CV1 was released, so I probably should undust it.

That wiki page is inadequate because it's only about using the Oculus Rift SDK version with GZ3Doom. I plan to use Valve's SDK, OpenVR, in our first HMD implementation in gzdoom, which requires an additional layer of software. You would need to install both the Oculus Rift software, and Valve's SteamVR, as described here http://www.roadtovr.com/how-to-use-ocul ... ther-mode/

[biospud]

OK, I got the 3D scene view, at least, looking correct in the wide side by side mode, by adjusting the projection matrix. And now both side-by-side modes use an optimally sized framebuffer.

Code: Select all

// AdjustViewports() is called from within FLGRenderer::SetOutputViewport(...)
void SideBySideBase::AdjustViewports() const
{
	// Change size of renderbuffer, and align to screen
	GLRenderer->mSceneViewport.width /= 2;
	GLRenderer->mSceneViewport.left /= 2;
	GLRenderer->mScreenViewport.width /= 2;
	GLRenderer->mScreenViewport.left /= 2;
}

https://github.com/cmbruns/gz3doom/blob ... 3d.cpp#L67

But I still do not love the current wide side-by-side appearance.

The 3D scene has the correct aspect ratio, which is good. But the weapon is too narrow. I would like to show the weapon twice as wide as it appears here, as it does in GZ3Doom https://github.com/cmbruns/gz3doom/wiki ... #vr_mode-3. But that GZ3Doom implementation was a nasty hack deep in the bowels of the weapon sprite rendering code.
And the status bar should be widened to occupy the full width of the viewport, because it is also too narrow. And the menus and title screens (not shown here) should be letterboxed into the center of each viewport.

[dpJudas]

This is tricky. The zdoom part that issues 2D drawing commands relies on SCREENWIDTH and SCREENHEIGHT. They always stay the same unless the resolution is changed and I have no idea how the code will react to you changing them on the fly.

If you want to experiment with it, change the OpenGLFrameBuffer Width and Height members to what you set the mScreenViewport size to. Something ala this in AdjustViewports:

GLRenderer->framebuffer->SetSize(GLRenderer->mScreenViewport.width, GLRenderer->mScreenViewport.height);

and then add this to the OpenGLFrameBuffer class in gl_framebuffer.h:

void SetSize(int w, int h) { Width = w; Height = h; } // What could possibly go wrong..

[biospud]

I think I'll postpone this particular problem with the wide side-by-side 3D mode for the moment. The time has come to sprint toward an HMD 3D mode, which contains a giant bag of related problems, the eventual solution to which might be the right answer for the wide-side-by-side mode also.

This past week, I was overjoyed to learn that we now have the big missing piece needed for phase 3 of my stereo 3d manifesto from last January (http://forum.zdoom.org/viewtopic.php?f=35&t=50349), namely this renderbuffer infrastructure. These side-by-side modes complete the proof-of-concept for phase 3, and also exercise a crucial element of phase 4 (HMD modes): custom scene framebuffer sizes.

Here's an outline of my current vision for the process flow for HMDs using OpenVR:

• * At Program Startup Time:
  • Detect whether OpenVR 3D mode is available
    Update VR mode menu accordingly
  * At Enter OpenVR Mode Time:
  • Detect whether HMD is connected
    • if not, issue warning and revert to mono mode...
    (Assuming HMD is connected:)
    Display a static "Put on HMD now" image to the screen. This will not be updated again, to avoid vsync mismatch stalling issues.
    Change size of render buffer to suit HMD, regardless of video mode setting
    Cache per-eye projection matrices, and per-eye incremental view matrices, which should not change.
    Ensure gzdoom is rendering without delays for vsync or anything else.
  * At Scene Render Time [gamestate GS_LEVEL|GS_TITLELEVEL]
  • Get latest predicted HMD pose from OpenVR API [VRCompositor.WaitGetPoses(...)]
    For each eye [left, right]:
    • Set up view parameters for this eye
      Activate scene renderbuffer
      Clear scene renderbuffer
      Render 3D scene
      Compose cross-hair into one quad in the 3D scene (alpha blended, no depth test, no depth write)
      Draw an optional laser site 3D line from weapon through cross-hair (translucent, yes depth test, no depth write)
      Draw HTC Vive controller models into 3D scene (yes depth test/write)
      Compose weapon sprite into another quad in the 3D scene (alpha blended, yes depth test, no depth write)
      Compose other 2D HUD items into another quad in the 3D scene (probably alpha blended, optional depth test, no depth write)
      Apply postprocessing (resolve multisampling, bloom, etc.)
      Apply "blend" effect (radiation/berserk/pickup-flash) over entire 3D scene
      Apply (invulnerability) "colormap" effect
      Store view into eye-specific renderbuffer
  * At Non-Scene screen->Update() Time [gamestate != GS_LEVEL|GS_TITLELEVEL]:
  • Get latest predicted HMD pose from OpenVR API [VRCompositor.WaitGetPoses(...)]
    For each eye [left, right]:
    • Set up view parameters for this eye
      Activate scene renderbuffer
      Clear scene renderbuffer
      Draw a custom non-uniform textured 360 panorama background, with a distinct horizon.
      Draw HTC Vive controller models into 3D scene (yes depth test/write)
      Compose 2D items into a quad in the 3D scene (yes depth test)
    Set up a timer to render again soon, if we would otherwise need to wait 30 milliseconds for the next tic update.
  * At Present Time:
  • [Do NOT apply sRGB gamma correction; OpenVR expects a photometrically linear texture]
    Submit final left-eye rendertexture to OpenVR API [VRCompositor.submit()]
    Submit final right-eye rendertexture to OpenVR API [VRCompositor.submit()]
    [OpenVR VRCompositor presents on it's own schedule]
  * At Exit OpenVR Mode:
  • Restore FGL_RenderBuffers size to match video mode
    Restore rendering to the screen

[dpJudas]

I tried playing around a little bit with changing the Width/Height of the framebuffer dynamically with the following code at the end of OpenGLFrameBuffer::Update:

Code: Select all

	Width = GetClientWidth();
	Height = GetClientHeight();
	GLRenderer->SetOutputViewport(nullptr);
	Begin2D(false);
	if (StatusBar != NULL) StatusBar->ScreenSizeChanged();

It almost worked. The zdoom part of the code does react to it and adjusts its rendering accordingly, but it doesn't recognize the aspect ratio properly. The function CheckRatio is responsible for doing this and my personal opinion is that the entire way it tries to deal with aspect ratios can never work well. To quote its own documentation:

Code: Select all

// Tries to guess the physical dimensions of the screen based on the
// screen's pixel dimensions. Can return:
// 0: 4:3
// 1: 16:9
// 2: 16:10
// 3: 17:10
// 4: 5:4
// 5: 17:10 (redundant)
// 6: 21:9

Such a simplistic model for describing the aspect ratio cannot work well for anything but the most naive cases. My personal favorite is #5 that states it is redundant, but no further explanation for why there's a need for a redundant entry.

Unfortunately I don't think that you can ever support HMD modes well without addressing this root issue of how zdoom handles aspect ratios. The hack here runs pretty deep, like SBarInfo's CommandAspectRatio where something maps from one table of aspect ratios to another table and then sets a truth, whatever that is: https://github.com/coelckers/gzdoom/blo ... .cpp#L1917.

Anyway, it really comes down to two choices I think:

1) Lie to zdoom's crappy implementation about what dimensions your eyes really have. This will stretch things. If your eye has a size that is close to one of the hacks, then it will hopefully not be too noticeable.

2) Fix the hack. This is of course the correct solution, but it is hard without a good understanding of what stuff like that CommandAspectRatio thing is up to. On the plus side, fixing this once and for all will help fix other issues such as the black bars in windowed mode.

[Graf Zahl]

There'S some really ugly issues caused by this, for example, ZDoom's blanket assumption that camera textures always have the same aspect ratio. It's really one huge mess that definitely needs to be fixed, but with large parts of the software renderer depending on this crap it's going to be very hard.

[dpJudas]

I could probably fix the software renderer part thanks to my experience from the truecolor branch. It is the HUD stuff like that sbarinfo that I don't know well enough.

The software renderer uses CheckRatio to find the aspect ratio. Unless the camera texture is exactly one of the magic 6 sizes it falls back to 4:3. Yet another example of why this function needs to be nuked from orbit.