OpenTK (OpenGL) + OculusWrap 1.6.0.0 SwapTextureSet Issue

Jun 15, 2015 at 9:53 PM
Hi!
I have been working on getting OculusWrap ready with OpenGL for three days but I'm having some problems with the SwapTextureSet at the moment.

As the topic says I'm using OpenTK (1.1.1589.5942) with the current OculusWrap version.
OculusWrap in ver 1.4.4. worked fine with OpenTK, cause the eye-textures were under app control and not shared like in 0.6.0.0 between graphics context and the ovr compositor service.

Each eyeTexture (rendertarget) has its own textureSet which has two textures for swapping in it. So far so good.

My implementation is almost the same like the C++ OculusTinyRoomGL version but it doesn't work.
For some reason the shared texture gets corrupt after the very first frame-rendering.
What happens is, that the very first draw to the shared eye textures succeeded but then fails for some unknown reason for the following frames continuously.

In my mirror window and through the rift I can see the first succeeded texture. When I move the rift I can see that the timewarp is applied to the first drawn texture continuously.
Everything (Scene rendering, GetTrackingState, etc.) is working perfectly but the swap texture fails.
I proved this through copying the pixeldata from the current eyetexture framebuffer as bmp to file each frame-rendering and I can see that the scene is drawn correctly.

Then I checked through GL.CheckFramebufferStatus() after binding to the new swapped texture (SetAndClearRendersurface) from the textureSet that there is a problem.
Following situation occurs:

first render(CurrentIndex++):
bothEyeFbo's -> CurrentIndex = 1 -> Check says: FramebufferComplete (this is good)

second render (CurrentIndex++):
bothEyeFbo's -> CurrentIndex = 0 -> Check says: FramebufferUnsupported ( not good )
...
This scenario proceeds to infinity, so CurrentIndex = 0 always returns FramebufferUnsupported

I also checked the Fbo's without bound depth buffer to it (only color texture from textureSet bound to it) and the same problem occurs.

FramebufferUnsupported means that the color format is not supported, but I explicitly specify in the SwapTextureSet creation that I want a RGBA format.

Now I'm at a point where I have no clue how to solve this. I only can guess what might be the problem.
Maybe the compositor service alters the texture data in a way that OpenTK can't understand. Cause the C++ OculusTinyRoomGL Version works fine.
I also checked in the C++ OculusTinyRoomGL Version if there is a Fbo binding issue and there is not (all FramebufferComplete).

I hope I'm not the only one who tried to get it working with OpenTK.
Unfortunately, I have to revert to OculusWrap 1.4.4.0 now. :(
Jun 16, 2015 at 12:37 PM
I produced the same problem scenario in the C++ version by not incrementing the CurrentIndex (always 0). Thus, the compositor locks the texture[0] every frame and can't be bound to a framebuffer/rendertarget.

So my guess is now that somehow the CurrentIndex of the SwapTextureSet got set to 0, regardless what is actually set.
Or there might be an Interopconversion error like wrong size calculation.

I had a look into the Wrap Src and couldn't find anything noticeable but I'm no expert in how to do proper marshalling and size calculation.
Coordinator
Jun 17, 2015 at 11:21 AM
Hi Andy1234,

Unfortunately I don't have any OpenTK sample to test against, so I have had to guess what was needed, based on how data was laid out in the C++ sample that came with the Oculus SDK. It's quite possible that there's a marshaling problem that needs to be addressed.

If there is a place where you can upload the test code, you have made, then I will be happy to take a look at it and determine if something in OculusWrap needs to be corrected.
Jun 18, 2015 at 11:54 AM
I put it on github: https://github.com/andy83/rift

Let me know if there's something unclear.
Coordinator
Jun 20, 2015 at 10:14 AM
After several hours of debugging, I finally found out what the problem is.

Troubleshooting the issue was a challenge for me, as my OpenGL skills are pretty rusty (I haven't used it in more than 10 years), but I found that after SubmitFrame is called, the Oculus SDK apparently takes ownership of texture that was just rendered, preventing you from accessing it.

Normally this won't be a problem, as you will increment the active texture, of the texture set, allowing the Oculus SDK to do whatever it wants with the previously active texture, while you work on the newly active texture set instead.

The SwapTextureSet class that was written to wrap the unmanaged SwapTextureSet class, was however not updating the unmanaged SwapTextureSet object, when the active texture set index was being incremented. This means that the Oculus SDK wasn't informed about which texture was set to active, causing it to still hang on to the texture, which in turn prevented OpenGL from accessing the same texture.

I have checked in the fixed code and will be creating a new release, shortly.

Thanks for providing the OpenGL sample code necessary for me to debug this issue!
Coordinator
Jun 20, 2015 at 10:31 AM
Ah, I almost forgot...

You need to uncomment the line:
// Pre-Increment SwapTextureIndex: First draw fails cause SubmitFrame locks texture0 anyway
eyeRenderTexture[eyeIndex].TextureSet.CurrentIndex++;
And comment out the line:
// Post-Increment SwapTextureIndex: First draw to texture0 succeeds
//eyeRenderTexture[eyeIndex].TextureSet.CurrentIndex++;
In your OpenGL example code, for it work.
Jun 20, 2015 at 12:23 PM
That's absolutely brilliant!

Thank you so much for the effort!

I updated the github project and it's working fine.

I'm really happy now to finally use it in my project.

Many thanks...
Jun 20, 2015 at 8:28 PM
I'm so sorry for bothering you again with another issue.

After changing back to the current version in my real project, the rendertargets are now working as expected.
But unfortunately my whole scene is upside down. In my SimpleDemo Project I didn't notice it as it was only a rotating cube, but in my real project the user is in a room.
First I thought they changed the rift coordinate system in the new version but after testing some settings with the new layer system I have that feeling that the compositor is ignoring the layerflags and type in the header part.

For example I changed the type to disabled and its not supposed to be shown but it does.
In your demo project I also changed the flag to TextureOriginAtBottomLeft and it's supposed to be upside down but unfortunately it's not.
So there must be an issue with the layer.

Swapping my viewmatrix by negative y-axis is not an option for me as my whole controllogic is going to be screwed up.

I hope this is not a big issue.
Coordinator
Jun 21, 2015 at 2:53 PM
I can see what the problem is.

When adding a new layer, a new LayerEyeFov is allocated and returned, such that it's possible to modify it. It's also added to a list, inside the Layers class, such that the list of layers can be converted into unmanaged memory, when SubmitFrame is called. This would have worked fine, except that the LayerEyeFov is defined as a struct and not a class, which means that the returned LayerEveFov isn't the same instance as is being converted to unmanaged memory.

I will have to create a set of wrapper classes for the LayerEyeFov and all the other layer classes as well, such that the object returned by adding a new layer, will be the same instance as is being converted into unmanaged memory.

I will let you know when I have a new version ready.
Coordinator
Jun 22, 2015 at 3:24 PM
I've created a new version, which should solve the problem with layers not getting properly marshaled.

Unfortunately I can't commit the code, as the CodePlex TFS server appears to be down.

I can't create a new release either, without being able to commit the code, so I don't know how long it will take before the project can be updated with the required changes.

I will let you know, as soon the changes are checked in.
Jun 22, 2015 at 9:00 PM
Allright many thanks in advance!
Coordinator
Jun 23, 2015 at 2:03 PM
The CodePlex TFS servers are up again.

The changes have been checked in and a new 1.6.0.2 release has replaced the previous 1.6.0.1 release.

I hope this solves the problems you have experienced :)
Marked as answer by MortInfinite on 6/27/2015 at 11:17 PM
Oct 2, 2015 at 11:56 AM
Hi,

Sorry for resurrecting this topic but it may be related...
I've update Andy's demo https://github.com/andy83/rift to SDK 0.7 (which went smoothly by the way) but after the update I'm experiencing one issue.
When changing HDM's Pitch by moving my head so that I look up, the rendered cube moves in the same direction as my eyes - which is totally not correct. When changing HMD's Yaw by looking left or right the cube is rendered correctly - it stays in the world where it was - correct.
On SDK 0.6 it's working correctly.

Any ideas what might be wrong? I can put the code to Github if needed.

Thanks in advance for looking into this. And by the way - great job with the wrapper - I've been searching for this for quite a while - thanks!
Coordinator
Oct 2, 2015 at 5:08 PM
The last time we saw that, I believe the problem was with the LayerEyeFov.Header.Flags being set to OVR.LayerFlags.None, instead of OVR.LayerFlags.TextureOriginAtBottomLeft.

If your LayerEyeFov.Header.Flags is set to OVR.LayerFlags.TextureOriginAtBottomLeft, try setting it to OVR.LayerFlags.None.
If your LayerEyeFov.Header.Flags is set to OVR.LayerFlags.None, try setting it to OVR.LayerFlags.TextureOriginAtBottomLeft.
(Basically try setting it to the opposite texture origin value than what is currently is).

Let me know if that solves it and if not, I can have a look at your code and see if I can figure out what went wrong :)
Oct 2, 2015 at 10:15 PM
Edited Oct 2, 2015 at 11:20 PM
Thanks m8 ;) It indeed works. I've changed LayerEyeFov.Header.Flags to OVR.LayerFlags.None - it was OVR.LayerFlags.TextureOriginAtBottomLeft before.
But what does that really mean? Is that correct that it's OVR.LayerFlags.None now? Why was it different in SDK 0.6 ? I guess it was correct in SDK 0.6, because OpenGL Texture coordinates in fact start from bottom left. So what's the story behind it?

BTW, here is the updated code:
https://github.com/Airstriker/OpenTKOculusDemo
Coordinator
Oct 3, 2015 at 5:07 AM
That is a very good question. Unfortunately I'm probably not the best person to answer that question, as I'm merely passing the values to the Oculus SDK.

Since the Oculus SDK started doing a lot more of the DirectX and OpenGL implementation themselves, we as developers need to do a lot more guessing. We could previously look in the source code of their SDK, to determine what they were doing, but most of their implementation is now done by their runtime library, which we don't have the code for.

Whenever there has been an update to the SDK, I've had to change the DirectX demo to accommodate some sort of change, like this one, without any explanation from the release notes of the Oculus SDK. I've come to the conclusion that the implementation underneath has probably changed, which requires everyone to change their use of the SDK as well, without being able to explain why.

You could try having a look at the Oculus forums (https://forums.oculus.com/) and see if anyone there has had the same experience with the Oculus 0.7 SDK.
Nov 11, 2015 at 4:11 PM
Edited Nov 11, 2015 at 4:30 PM
I've found the universal solution to the problem with inverted pitch, roll, yaw problem:
https://forums.oculus.com/viewtopic.php?t=17841

Basically, all you need to do is pre and post multiply the rotation matrix by the appropriate scaleMatrix:
Quaternion rotationQuaternion = layerFov.RenderPose[eyeIndex].Orientation.ToTK();
Matrix4 rotationMatrix = Matrix4.CreateFromQuaternion(rotationQuaternion);

Matrix4 scaleMatrix = Matrix4.CreateScale(1f, 1f, -1f); //X=right, Y=up, Z=in
rotationMatrix = scaleMatrix * rotationMatrix * scaleMatrix;
I've added this to the OpenTKOculusDemo.