I feel like this is sort of how foveated rendering works now in VR with eye tracking. Rendering only what’s directly in the center of your view at full resolution to reduce the rendering workload and boost performance. Since you’re not going to actually view all that a full resolution anyway in your peripheral vision.
It should be a lot harder than foveated rendering. For foveation you need to know the position of the eye, but for dof you need the shape of the lense inside the eye.
You could try estimating the focus based on what the eyes together point at, but the eyes don’t actually focus in perfect match with their stereo vision.
And then if you do manage to track it, you need to undo the eyes lense too, since it’s naturally trying to blur the screen away from being in focus.
Currently vr just ignores it and the eye just doesn’t focus like it naturally would, but if you wanna have say the ability to focus between two overlapping things with the other blurring out of the way, you need that lense input, so you need the eye to focus, so you need to let the lense squeeze while the screen stays in focus.
Tracking lenses seems so hard I wouldn’t be surprised if we get light field displays first. Which sidestep the whole problem and are really the right way to do it. Proper sharpness and blur on a passive screen even without tracking. For a good looking blur you probably only need a handful angular pixels, since you know the exact location the eye is at.
Would need (flat surface) holography style screens to really trick the eyes and fully simulate long range depth rather than simulating it. A proper dynamic holographic screen wouldn’t even need fancy lenses (unless you need to resize perhaps). Most current light field-ish displays are just doing mirror/prism arrays over pixels, which looks neat at a distance, but don’t have the resolution to trick the eye (see: the external eye screen on the Vision Pro).
Ignoring polarization and pretending there’s only 3 colors, a light field display should be equivalent given enough resolution.
A few angular pixels don’t give you stereo vision or parallax or anything like that, but if you make them show a “pre blurred” image based on angle you can probably get extremely close to natural out of focus. Basically in the center you present nornal pixels, then off-axis you have pixels that are colored with their own blur applied. So when defocussing, instead of capturing quadratically more angular pixels, you’re capturing just a few more that are each averages of what all those pixels would have been.
This only has any chance of working because the eye is at a known location and defocusses in a known basic way.
Even then you’d need to account for the physical movement of the pupil when the eye turns, etc. Maybe the lense layer could be shifted in response to keep the angular pixel distribution pointed right.
There’s also the processing problem, you’d need to render way more than is currently done.
Either way it’s nowhere near viable at the moment, should be multiple decades away at minimum.
PS:
Technically the 3ds had a light field display with 2 angular pixels. You can get a lot done with low resolution if you force the eyes to be in a specific location.
Yeah, VR doesn’t need to fake it. Foveated rendering does fake it to an extent, but it’s purely at a detriment to image quality, but it’s out-of-focus anyway, so that doesn’t really matter, but the performance increase does.
I feel like this is sort of how foveated rendering works now in VR with eye tracking. Rendering only what’s directly in the center of your view at full resolution to reduce the rendering workload and boost performance. Since you’re not going to actually view all that a full resolution anyway in your peripheral vision.
https://en.wikipedia.org/wiki/Foveated_rendering
It should be a lot harder than foveated rendering. For foveation you need to know the position of the eye, but for dof you need the shape of the lense inside the eye.
You could try estimating the focus based on what the eyes together point at, but the eyes don’t actually focus in perfect match with their stereo vision.
And then if you do manage to track it, you need to undo the eyes lense too, since it’s naturally trying to blur the screen away from being in focus.
Currently vr just ignores it and the eye just doesn’t focus like it naturally would, but if you wanna have say the ability to focus between two overlapping things with the other blurring out of the way, you need that lense input, so you need the eye to focus, so you need to let the lense squeeze while the screen stays in focus.
Tracking lenses seems so hard I wouldn’t be surprised if we get light field displays first. Which sidestep the whole problem and are really the right way to do it. Proper sharpness and blur on a passive screen even without tracking. For a good looking blur you probably only need a handful angular pixels, since you know the exact location the eye is at.
Would need (flat surface) holography style screens to really trick the eyes and fully simulate long range depth rather than simulating it. A proper dynamic holographic screen wouldn’t even need fancy lenses (unless you need to resize perhaps). Most current light field-ish displays are just doing mirror/prism arrays over pixels, which looks neat at a distance, but don’t have the resolution to trick the eye (see: the external eye screen on the Vision Pro).
Ignoring polarization and pretending there’s only 3 colors, a light field display should be equivalent given enough resolution.
A few angular pixels don’t give you stereo vision or parallax or anything like that, but if you make them show a “pre blurred” image based on angle you can probably get extremely close to natural out of focus. Basically in the center you present nornal pixels, then off-axis you have pixels that are colored with their own blur applied. So when defocussing, instead of capturing quadratically more angular pixels, you’re capturing just a few more that are each averages of what all those pixels would have been.
This only has any chance of working because the eye is at a known location and defocusses in a known basic way.
Even then you’d need to account for the physical movement of the pupil when the eye turns, etc. Maybe the lense layer could be shifted in response to keep the angular pixel distribution pointed right.
There’s also the processing problem, you’d need to render way more than is currently done.
Either way it’s nowhere near viable at the moment, should be multiple decades away at minimum.
PS:
Technically the 3ds had a light field display with 2 angular pixels. You can get a lot done with low resolution if you force the eyes to be in a specific location.
Yeah, VR doesn’t need to fake it. Foveated rendering does fake it to an extent, but it’s purely at a detriment to image quality, but it’s out-of-focus anyway, so that doesn’t really matter, but the performance increase does.