I don't know what the sample you pasted is actually doing, but somehow I think it's neglecting the extra *1.2f that YaspectMul was doing. Change that to a simple constant, and it seems to work better.
That being said, I managed to make it work even closer to what GZDoom was doing. It executes two scalings - one is before the rotation one is after, and with the one before the rotation being customizable I've found that I was able to input any non-negative value and it worked.
If you are planning to implement mirrors though you're going to have to turn off backface culling.
I've simplified the code down to this:
Code: Select all
	static bool bDidSetup = false;
	if (!bDidSetup)
	{
		InitGLRMapinfoData();
		bDidSetup = true;
	}
	float pixelstretch = (glset.pixelstretch) ? glset.pixelstretch : 1.2;
	float ratio = WidescreenRatio;
	float fovratio = (WidescreenRatio >= 1.3f) ? 1.333333f : ratio;
	float fovy = (float)(2 * DAngle::ToDegrees(atan(tan(FieldOfView.Radians() / 2) / fovratio)).Degrees);
	TriMatrix worldToView =
		TriMatrix::scale(1.0f, 1.2f, 1.0f) *
		TriMatrix::rotate((float)ViewPitch.Radians(), 1.0f, 0.0f, 0.0f) *
		TriMatrix::rotate((float)(ViewAngle - 90).Radians(), 0.0f, -1.0f, 0.0f) *
		TriMatrix::scale(1.0f, pixelstretch, 1.0f) *
		TriMatrix::swapYZ() *
		TriMatrix::translate((float)-ViewPos.X, (float)-ViewPos.Y, (float)-ViewPos.Z);
	WorldToClip = TriMatrix::perspective(fovy, ratio, 5.0f, 65535.0f) * worldToView;
Though it still needs testing before a merge. So far, it works, though.
Also, the GLData setup should probably be done elsewhere in the code, as well.