I believe I know how the matrices work (EDIT: sort of, as I was proven later), but I do not know Unity. If we figure out, what world unit it uses or how the distance I calculated correlate to the real IPD, I might be able to calculate the projection matrix with the rotation for the displays. If you know how to change it in the game (which your previous post suggests you know), you might be able to make it work without parallel projection.
1 Like
Yep, Iâm just directly hooking Camera.GetStereoProjectionMatrix (with harmony) and changing the values.
This results in an effect that is still slightly offset, but aligns (mostly) in both eyes so itâs less disorienting. It would also hypothetically work on any unity game if we can get the matrices adjusted correctly.
1 Like
Bluetooth adds latency.
1 Like
Yes. I forgot to post an update here, the problem was indeed the Bluetooth headset.
2 Likes
Hey, sorry for the late reply (was sleeping lol).
This project depends on CustomUI and Harmony
Also, in the HMD I have my IPD set to near 65 (although I canât confirm what it was when I took those readings, so if you want I can adjust the IPD and regrab the readings).
Edit: It appears to me that the IPD set in the headset actually does not impact these numbers at all.
IPD 60.3
Eye: Right
0.64759 0.00000 0.12824 0.00000
0.00000 0.78750 0.00000 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
Eye: Left
0.64759 0.00000 -0.12824 0.00000
0.00000 0.78750 0.00000 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
IPD 72.8
Eye: Right
0.64759 0.00000 0.12824 0.00000
0.00000 0.78750 0.00000 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
Eye: Left
0.64759 0.00000 -0.12824 0.00000
0.00000 0.78750 0.00000 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
Iâm about to connect my vive and see what the readings are for itâŚ
Here are the readings with my vive connected (checked with multiple different IPD readings as well, also the same regardless of IPD):
Eye: Right
0.76389 0.00000 0.05727 0.00000
0.00000 0.68750 -0.00463 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
Eye: Left
0.76389 0.00000 -0.05581 0.00000
0.00000 0.68750 0.00140 0.00000
0.00000 0.00000 -1.00020 -0.20002
0.00000 0.00000 -1.00000 0.00000
2 Likes
What bothers me also is the Vive matrices which are slightly different. They both show the frustum of the same volume, but shifted differently (though the difference is not big), and not only horizontally (as stereo projection would suggest), but also vertically!
Vive left "remat" projection matrix:
[[ 0.76389 0. -0.05581 0. ]
[ 0. 0.6875 0.0014 0. ]
[ 0. 0. -1.0002 -0.20002]
[ 0. 0. -1. 0. ]]
l=-0.138215, r=0.123603, b=-0.145251, t=0.145658, n=0.100000, f=1000.100000
projection dimens: horiz=0.261818, vert=0.290909
projection offset: horiz=-0.007306, vert=0.000204
Vive right "remat" projection matrix:
[[ 0.76389 0. 0.05727 0. ]
[ 0. 0.6875 -0.00463 0. ]
[ 0. 0. -1.0002 -0.20002]
[ 0. 0. -1. 0. ]]
l=-0.123412, r=0.138406, b=-0.146128, t=0.144781, n=0.100000, f=1000.100000
projection dimens: horiz=0.261818, vert=0.290909
projection offset: horiz=0.007497, vert=-0.000673
As if your Vive was somehow ârecalibratedâ and used slightly ârolledâ view. Anyway, as the values seem to be totally arbitrary, I am not sure how to interpret them (especially considering they do not change with the IPD reading change).
On the other hand, I have built your project, but cannot patch UnityEngine.dll with the IPA. It says:
Patching UnityEngine.dll... ERROR: Oops! This should not have happened.
System.Exception: Could not find any entry type!
at IPA.Patcher.PatchedModule.Patch()
at IPA.Program.Install(PatchContext context)
[Press any key to quit]
And then I cannot start the game, nor âunpatchâ it. I had to revert to the good binary from Steam (check files integrity - at least I hope it was what fixed that). I assume that if I do not patch it, your plugin does not run.
When I finally run the game (with parallel projection OFF, and Normal FOV) I noticed that the geometry seems to be correct, only some effects are off (I never run BS in Pimax before, so I was not aware what the issue was exactly). This would suggest that the fundamental transformations must be correct, but the post processing effects (which may, or may not be done in 3D) somehow got screwed up.
I will try to debug some Unity VR demo to get a better grasp on the rendering pipeline.
3 Likes
Hm⌠I did do the gear vr lens swap on my vive a long time ago, and now that I think about it there was some config file that I had to replace to adjust the distortion profile. Could this have anything to do with it?
Also, you want to patch the game with IPA; not UnityEngine.dll. After doing so you simply put the plugins I posted above into the Plugins folder in your Beat Saber directory and you should be good to go.
If you use one of the mod installers it will automatically setup everything for you and give you a nice interface for downloading the dependencies I posted above as well.
https://github.com/lolPants/modsaber-installer/releases
Also, youâre correct; the game its self looks absolutely fine and only certain effects are broken without parallel projections (specifically a few lighting components, and mirrored effects). The only reason I bring up this GetStereoProjectionMatrix function is because when I looked at the source code to Beat Saber, I realized every effect that uses the matrices from this function is broken.
2 Likes
I only did what the wiki on github said: drag & drop the game executable on IPA.exe. This action seems to copy two DLLs and one XML from IPA folder to game Managed folder and then do some magic on UnityEngine.dll (and fails). I also noticed that UnityEngine.CoreModule.dll has different timestamp, from all other Unity DLLs in the same directory. But when I run âverify files integrityâ from the Steam client, it said, all was OK 
1 Like
Yeah that should work, not sure if a specific version of IPA is required for Beat Saber since itâs running on a slightly older version of unity, Iâve just used the version provided by the various mod installers that exist for the game (I believe its like 2018.1.x).
1 Like
Apparently it has been fixed in the git repo, so rebuilding IPA locally solved the failing patch. Now it seem I have it patched, but the game does not seem to be very stable, sometimes even does not start. I have to restart Steam and SteamVR in order to execute it successfully. So I am still not there.
1 Like
Try running the game with the --verbose launch option, this will pop up a console window which can often help to debug common problems.
The other place to check is %appdata%\..\LocalLow\Hyperbolic Magnetism\Beat Saber, thereâs a file there called output_log.txt which often has helpful information in it.
2 Likes
I am now able to run the game with your plug-in loaded. Now I have another question - is there an easy way to log or dump the data from the hook?
I tried to run the game in the debugger, but apparently this didnât work.
1 Like
I donât know of any way to use a debugger, I generally just use System.Console.WriteLine and read the output from the --verbose console window, or if you want it to log to the output_log.txt file use the Unity Debug.Log function.
2 Likes
@brian91292 and @risa2000, I just wanted to say thanks a lot for trying to fix this issue. I was playing it on my 5K+ for the first time yesterday and was slightly annoyed that the experience is so close to perfect!
@brian91292 You mentioned earlier that the BeatSaber devs are aware of the issue. Know if theyâre planning to spend any time fixing it?
2 Likes
No clue, all I know is I mentioned issue directly to one of the devs and he acknowledged that he would pass the info on.
1 Like
Backer or Pre order? 
Backer 
One of the last UK deliveries
1 Like
Unfortunately I think we still have some waiting on there delivery.
Congrats!
@anon74848233 please add to earlybackers
@brian91292 I have progressed a bit (kind of) with understanding the problem:
- I checked my projection matrices in BS and they are the same as yours (and do not change when changing IPD on the headset)
- Unity uses metric system so 1.0 = 1.0 meter.
- As you already did, I also noticed that left view is ok, but right is shifted.
I assume the last point happens because there is a discrepancy somewhere between the âeffect coordinatesâ and the âgeometry coordinatesâ. It looks like the devs âtweakedâ something in the game to get the left eye right, but forgot to check the right eye and the thing they tweaked was not the right one.
1 Like
Seems youâre right, I actually donât think the issue is with these projection matrices at all at this point⌠the issue seems to be related to how they are being used.
Iâve continued trying to narrow down the problem, and the lighting issue seems to stem from TubeBloomPrePassLight.FillMeshData
public override void FillMeshData(int lightNum, Vector3[] vertices, Color[] colors, Vector4[] viewPos, Matrix4x4 viewMatrix, Matrix4x4 projectionMatrix, float lineWidth)
{
float y = -this._length * this._center;
float y2 = this._length * (1f - this._center);
Matrix4x4 localToWorldMatrix = this._transform.localToWorldMatrix;
Vector3 point = localToWorldMatrix.MultiplyPoint3x4(new Vector4(0f, y, 0f));
Vector3 point2 = localToWorldMatrix.MultiplyPoint3x4(new Vector4(0f, y2, 0f));
Vector3 v = viewMatrix.MultiplyPoint3x4(point);
Vector3 v2 = viewMatrix.MultiplyPoint3x4(point2);
Vector4 a = projectionMatrix * new Vector4(v.x, v.y, v.z, 1f);
Vector4 a2 = projectionMatrix * new Vector4(v2.x, v2.y, v2.z, 1f);
bool flag = a.x >= -a.w;
bool flag2 = a2.x >= -a2.w;
if (!flag && !flag2)
{
for (int i = 0; i < 4; i++)
{
vertices[lightNum * 4 + i] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t = (-a.w - a.x) / (a2.x - a.x + a2.w - a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t);
}
flag = (a.x <= a.w);
flag2 = (a2.x <= a2.w);
if (!flag && !flag2)
{
for (int j = 0; j < 4; j++)
{
vertices[lightNum * 4 + j] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t2 = (a.w - a.x) / (a2.x - a.x - a2.w + a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t2);
}
flag = (a.y >= -a.w);
flag2 = (a2.y >= -a2.w);
if (!flag && !flag2)
{
for (int k = 0; k < 4; k++)
{
vertices[lightNum * 4 + k] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t3 = (-a.w - a.y) / (a2.y - a.y + a2.w - a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t3);
}
flag = (a.y <= a.w);
flag2 = (a2.y <= a2.w);
if (!flag && !flag2)
{
for (int l = 0; l < 4; l++)
{
vertices[lightNum * 4 + l] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t4 = (a.w - a.y) / (a2.y - a.y - a2.w + a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t4);
}
flag = (a.z <= a.w);
flag2 = (a2.z <= a2.w);
if (!flag && !flag2)
{
for (int m = 0; m < 4; m++)
{
vertices[lightNum * 4 + m] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t5 = (a.w - a.z) / (a2.z - a.z - a2.w + a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t5);
}
float num = 0.0001f;
flag = (a.z >= -a.w - num);
flag2 = (a2.z >= -a2.w - num);
if (!flag && !flag2)
{
for (int n = 0; n < 4; n++)
{
vertices[lightNum * 4 + n] = Vector3.zero;
}
return;
}
if (flag != flag2)
{
float t6 = (-a.w - a.z) / (a2.z - a.z + a2.w - a.w);
this.ClipPoints(ref a, ref a2, ref v, ref v2, flag, t6);
}
Vector3 vector = a / a.w;
Vector3 a3 = a2 / a2.w;
vector.x = vector.x * 0.5f + 0.5f;
vector.y = vector.y * 0.5f + 0.5f;
vector.z = 0f;
a3.x = a3.x * 0.5f + 0.5f;
a3.y = a3.y * 0.5f + 0.5f;
a3.z = 0f;
Vector3 vector2 = a3 - vector;
Vector3 vector3 = new Vector3(-vector2.y, vector2.x, 0f);
vector3.Normalize();
vector3 *= lineWidth;
int num2 = lightNum * 4;
vertices[num2] = vector - vector3;
vertices[num2 + 1] = vector + vector3;
vertices[num2 + 2] = a3 + vector3;
vertices[num2 + 3] = a3 - vector3;
Color color = this._color * this._intensityMultiplier;
colors[num2] = color;
colors[num2 + 1] = color;
colors[num2 + 2] = color;
colors[num2 + 3] = color;
viewPos[num2] = v;
viewPos[num2 + 1] = v;
viewPos[num2 + 2] = v2;
viewPos[num2 + 3] = v2;
}
The Render function below eventually winds up calling FillMeshData with the projection matrix being the halfway point between the left and right eye:
public void Render(Camera camera, BloomPrePassParams bloomPrePassParams, RenderTexture dest, out Vector2 textureToScreenRatio)
{
bool sRGBWrite = GL.sRGBWrite;
GL.sRGBWrite = false;
Matrix4x4 projectionMatrix;
if (camera.stereoEnabled)
{
Matrix4x4 stereoProjectionMatrix = camera.GetStereoProjectionMatrix(Camera.StereoscopicEye.Left);
Matrix4x4 stereoProjectionMatrix2 = camera.GetStereoProjectionMatrix(Camera.StereoscopicEye.Right);
projectionMatrix = this.MatrixLerp(stereoProjectionMatrix, stereoProjectionMatrix2, 0.5f);
}
else
{
projectionMatrix = camera.projectionMatrix;
}
textureToScreenRatio.x = Mathf.Clamp01(1f / (Mathf.Tan(bloomPrePassParams.fov.x * 0.5f * 0.0174532924f) * projectionMatrix.m00));
textureToScreenRatio.y = Mathf.Clamp01(1f / (Mathf.Tan(bloomPrePassParams.fov.y * 0.5f * 0.0174532924f) * projectionMatrix.m11));
projectionMatrix.m00 *= textureToScreenRatio.x;
projectionMatrix.m02 *= textureToScreenRatio.x;
projectionMatrix.m11 *= textureToScreenRatio.y;
projectionMatrix.m12 *= textureToScreenRatio.y;
Matrix4x4 worldToCameraMatrix = camera.worldToCameraMatrix;
RenderTexture temporary = RenderTexture.GetTemporary(bloomPrePassParams.textureWidth, bloomPrePassParams.textureHeight, 0, RenderTextureFormat.RGB111110Float, RenderTextureReadWrite.Linear);
Graphics.SetRenderTarget(temporary);
GL.Clear(true, true, Color.black);
this.RenderAllLights(worldToCameraMatrix, projectionMatrix, bloomPrePassParams.linesTexture, bloomPrePassParams.linesWidth, bloomPrePassParams.linesFogDensity, bloomPrePassParams.lineIntensityMultiplier);
RenderTexture temporary2 = RenderTexture.GetTemporary(bloomPrePassParams.textureWidth >> bloomPrePassParams.downsample, bloomPrePassParams.textureHeight >> bloomPrePassParams.downsample, 0, RenderTextureFormat.RGB111110Float, RenderTextureReadWrite.Linear);
this._kawaseBlurRenderer.DoubleBlur(temporary, dest, bloomPrePassParams.bloom1KernelSize, bloomPrePassParams.bloom1Boost, temporary2, bloomPrePassParams.bloom2KernelSize, bloomPrePassParams.bloom2Boost, bloomPrePassParams.downsample);
RenderTexture.ReleaseTemporary(temporary);
this._additiveMaterial.SetFloat(this._alphaID, bloomPrePassParams.bloom2Alpha);
Graphics.Blit(temporary2, dest, this._additiveMaterial);
RenderTexture.ReleaseTemporary(temporary2);
GL.sRGBWrite = sRGBWrite;
}
Anyway, I donât think I can really be of any more help on the issue unless you need some help understanding something about Beat Saber specifically thatâs unrelated to all this projection matrix stuff lol.