Stereo 3D stuff

[biospud]

I have an Oculus DK2, although I'm not sure if it even works anymore after they released the consumer version. That's about the only thing I have. But I can prepare FGLRenderBuffers and the eye handling code to run like I described. I don't have any plans this weekend, so sure, I'll be around.

Thanks. Prepare what you can, and I'll take it from there, starting on Friday evening USA eastern time. There is a good chance that the DK2 would work with a future VR 3D mode. I'm sketching out the procedural flow for HMD VR modes now, but I won't finalize it until we finish reimplementing the already published modes for the upcoming release. I'm curious whether the OpenVR API would work with the DK2. If you have the time and inclination, I encourage you to install SteamVR and try configuring it for the DK2. If that works, then you would have the capability to help implement and test the future OpenVR mode. If that does not work, it would be possible to create another mode using the Oculus Rift SDK, a la GZ3Doom, but I'm more motivated to focus on OpenVR at the moment, which might work for both Rift (CV1) and Vive.

I've expanded the process flow for the new way below. I identified seven locations in the control flow where 3D-mode-specific hooks may be needed. We only need to do six of them now. The seventh is reserved for future HMD modes.

I think the proposed mBuffers->BindEyeTexture() method might require two arguments: an eye index, and an OpenGL texture unit index.

Time sequence for non-HMD 3D rendering in gzdoom:

• * At Program-Startup Time:
  • Detect whether each 3D mode is available
    • gl_renderbuffers == true?
      other mode-specific requirements?
    Update VR mode menu accordingly
  * At Enter 3D Mode Time: [vr_mode CVAR changes]
  • Double check whether mode is available
    if not, issue warning and revert to mono mode...
    for example, quad-buffered 3D mode needs a 120Hz full-screen display mode on non-Quadro nvidia boards
    Allocate Left renderbuffer, if necessary
  * At Scene Render Time [FGLRenderer::RenderViewpoint()][gamestate == GS_LEVEL|GS_TITLELEVEL]
  • Set up view parameters for left eye view
    Activate scene renderbuffer
    Clear scene renderbuffer
    Render 3D scene
    Apply postprocessing (resolve multisampling, bloom, etc.)
    Apply (invulnerability) "colormap" effect
    Store final Left-eye view into Left renderbuffer [mBuffers->BlitToEyeTexture(0)]
    Set up view parameters for right eye view
    Activate scene renderbuffer
    Clear scene renderbuffer
    Render 3D scene
    Apply postprocessing (resolve multisampling, bloom, etc.)
    Apply (invulnerability) "colormap" effect
    [mBuffers->BlitToEyeTexture(1)]
  * At Non-Scene screen->Update() Time [gamestate != GS_LEVEL|GS_TITLELEVEL]:
  • (Do nothing for now, but future HMD modes need to do something clever here)
  * At Present Time: [FGLRenderer::CopyToBackbuffer()]
  • Compose 2D HUD Elements onto left eye view [mBuffers->BindEyeFB(0)]
    Compose 2D HUD Elements onto right eye view [mBuffers->BindEyeFB(1)]
    Compose left and right eye textures, using a mode-specific present shader
    mBuffers->BindEyeTexture(0, 0) // 2 arguments: buffer index, and OpenGL texture unit
    mBuffers->BindEyeTexture(1, 1) // 2 arguments: buffer index, and OpenGL texture unit
    load mode-specific shader program
    render
  * At Exit Mode Time: [vr_mode CVAR changes]
  • Undo mode-specific trickery
    clear GL_BACK, GL_BACK_LEFT, GL_BACK_RIGHT buffers in case of quad stereo
    restore GL_BACK target in case of quad stereo
  * At Mode Destruction Time:
  • Delete any locally acquired OpenGL resources

[dpJudas]

I've finished the code creating eye textures and making the scene render to it.

As a proof of concept, I've written a Present function for Stereo3DMode that copies the eye textures to the back buffer (https://github.com/dpjudas/zdoom/blob/e ... 3d.cpp#L83). It generates this output:



As can be seen on the image, the 2D part only gets rendered to the first eye texture.

Graf: is there an easy way to get it to draw all the 2D twice? Calling m2DDrawer->Flush() for each eye did not do the trick (see https://github.com/dpjudas/zdoom/blob/e ... s.cpp#L385).

[biospud]

It looks like m2DDrawer->Flush() draws its contents then clears its data:
https://github.com/dpjudas/zdoom/blob/e ... r.cpp#L493
We probably need to factor out a new F2DDrawer::FlushNoClear() method for this purpose.

[Graf Zahl]

Until now there was no reuse here so the Clear was part of the Flush.
I recomment splitting it into a Draw() and a Clear() function.

[dpJudas]

Splitting it into Draw and Clear did the trick. Thanks.

I noticed two issues in the process we need to deal with though:

1) Rendering the 2D twice like this will not work for a HMD mode because the 2DDrawer uses scissors. The 2D needs to be drawn with a perspective matrix in this case, which scissor clips can't handle. The only blend mode used seem to be standard alpha blend (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA). If that is correct, we can solve this problem by rendering the 2D to a texture and pass that on to the eye present.

2) Screen wipes. OpenGLFrameBuffer::Wipe* needs to be made aware of eye buffers. I wonder how to even do some of those effects with HMD output.

[Graf Zahl]

1. That's a problem. It uses scissors because it offers more precision when cutting of parts of the source graphic, but I guess that can be altered if the destination device cannot deal with it.
2. It can use any render style. The 2D texture drawer uses gl_SetRenderStyle to decode the info being passed to it.
3. Wipes are a problem indeed. I vote for the simple solution: Disable them if such a device is selected. That's really not worth complicating the code. The wipes probably won't look good anyway there.

[dpJudas]

Alright, so using a 2D texture is out. I'll leave this part of the equation to be solved by biospud when doing the HMD part. Maybe it can use plane clipping instead or something. We luckily don't need it for this release.

Disabling the wipes sounds good to me.

[Edward-san]

As can be seen on the image, the 2D part only gets rendered to the first eye texture.

Isn't there an option to display it on the second eye instead? If somebody finds more comfortable the right eye...

[biospud]

1. That's a problem. It uses scissors because it offers more precision when cutting of parts of the source graphic, but I guess that can be altered if the destination device cannot deal with it.
2. It can use any render style. The 2D texture drawer uses gl_SetRenderStyle to decode the info being passed to it.
3. Wipes are a problem indeed. I vote for the simple solution: Disable them if such a device is selected. That's really not worth complicating the code. The wipes probably won't look good anyway there.

1. The HMD solution for 2D graphics is a two step process: 1) Render the whole batch of 2D elements to a rendertexture; 2) Compose that rendertexture into the 3D scene as an alpha-blended projected quad. That's what I do routinely in GZ3Doom.

3. In GZ3Doom Oculus Rift mode I eventually disabled wipes, precisely because it was a complicated hassle.

[dpJudas]

1. The HMD solution for 2D graphics is a two step process: 1) Render the whole batch of 2D elements to a rendertexture; 2) Compose that rendertexture into the 3D scene as an alpha-blended projected quad. That's what I do routinely in GZ3Doom.

If I understood Graf correctly, the 2D rendering might use other (OpenGL) blending modes than plain alpha blending. In those situations starting out with a transparent rendertexture won't work as the blend rules need data from the scene behind the quad.

[dpJudas]

Pull request with my changes: https://github.com/coelckers/gzdoom/pull/96

It adds eye buffers to FGLRenderBuffers and hooks up all the current stereo modes to use them. Should be a better starting point for biospud - maybe with some luck most of the modes are already working with this patch.

[Graf Zahl]

At least the invulnerability effect looks correct now with the colormasked modes. This was horribly broken in older builds.

[dpJudas]

Yes, that part looks good now. I think it failed before because the inverting math needs all three color channels intact.

[Graf Zahl]

I meant before the colormap refactor. It wrote the inverted stuff directly into the color channels and it was totally off. This is the first time that part looks ok to me.

[biospud]

Wow! Great work dpJudas. My first quick run through of all the 3D stereo modes, using my various 3D eyewear, worked flawlessly. I tested all the 3D modes: Mono, Left, Right, Green/Magenta, Red/Cyan, Yellow/Blue, and Quad-Buffered; including conditions with Invulnerability and Radiation effects. The anaglyph modes actually look great under invulnerability, because they are best suited to scenes with a bright low-contrast background, rather the the usual dark gloomy doom environment. Already this is good enough that I don't think we need to block the upcoming release for the stereo 3D modes. It seems I won't need two days of work here after all...

After more careful testing, I did find one bug. But it's a bug that may have always been present. I started sketching a test plan for verifying the correctness of the 3D modes at https://github.com/cmbruns/gz3doom/wiki ... -test-plan
Eventually I will include some annotated screen shots there, so folks without eyewear can check the basic correctness of the 3D modes.

The one bug I found relates to quad buffered stereo. If I start gzdoom in quad stereo mode by including "+vr_mode 7" on the command line, the doom screen remains totally black, which is horrible. On the other hand, vr_mode 7 works correctly if I wait and activate it after beginning game play in gzdoom. It's possible this behavior is unchanged from previous versions.

If I understood Graf correctly, the 2D rendering might use other (OpenGL) blending modes than plain alpha blending. In those situations starting out with a transparent rendertexture won't work as the blend rules need data from the scene behind the quad.

Even if the blending ends up not-quite-right, I believe this two-pass approach is best. I propose we wait and see what blending troubles occur, and then form a plan of attack. Hmm. This might explain some confusion I experienced with the cross-hair blending in GZ3Doom.

I was surprised to notice that dpJudas did not need to create any new shaders to get this all working. I now realize that we might not need new shaders to implement all of the 3D modes I already implemented in GZ3Doom. But I believe custom present shaders could be the best approach for some other modes I never implemented (interleaved modes), and for refinements to the anaglyph modes, to use more subtle color arithmetic than this rather crude masking approach we have been using. It would be nice if the anaglyph 3D modes would suck a bit less (i.e. less crosstalk, and better brightness, especially through the red and blue lenses). I was sorta wishing that dpJudas would also create an example of how to create and use a custom present shader, but perhaps I'll need to figure that out for myself.

So the next tasks for me are:
* Further characterize the quad-buffered 3D mode black screen bug, and fix it.
* Use this wonderful new renderbuffer infrastructure to implement side-by-side 3D modes (for AFTER the upcoming release)
* Finish polishing my outline for the future OpenVR HMD 3D mode control flow, so I can share it here, while my new best friend dpJudas might still be paying attention
* Implement all of the many 3D modes that my awesome 2012 Samsung 3D TV supports, especially row-interleaved stereo, because that's actually a useful one