Inspired by a comment in my previous post, thought I'd write up something about VR. Yes, it's pretty cool, but it's a long way to mainstream acceptance. I see a few major challenges, of which few have been overcome. Note, my knowledge on this matter is incomplete, so I'd certainly recommend you do your own research. Feel free to add to this discussion in the comments.
Frame rate, display technology and motion tracking. Let's start with the good news. Motion tracking on current-gen headsets is certainly "good enough". As is the frame rate, at 90 Hz. Could things be better? Sure. The ideal frame rate is somewhere between 144 Hz and 200 Hz. But for now, 90 Hz is certainly good enough. The bad news? Some games and systems can actually target a smooth 90 fps. Also, they don't offer variable refresh, so every frame drop effective is 45 fps. We have seen PS VR games target 60 fps, often with dramatic compromise to image quality - simply not good enough.
OLED technology is also perfect for VR. With its infinite contrast ratio and lightning fast switching latencies, we are certainly not lacking image quality. Sure, things can keep improving - to meet the Rec.2020 colour space and HDR1000 standards - but I'd say OLED today is good enough. Unfortunately, some Windows Mixed Reality VR headsets use IPS, which is not good enough for a VR experience due to the high black levels. On an OLED display, black is simply black as the each pixel can turn off individually; while with IPS LCD displays, the backlight shines through even at black. As a result, black is really more like a dark grey.
Resolution. This is the biggest issue with current gen VR headsets. The dreaded screen door effect, where you can see the pixels. This totally breaks immersion.
Contrary to popular belief, resolution is not just a matter of screen size, but also viewing distance. While the VR displays are 2.5" in size, the viewing distance from eye to it is also tiny. We know that the average human eye requires 60 pixels per degree.
The calculation works out to requiring 4,000 dpi, or around 9000x9000 pixels per eye. If you're wondering, I got this number from an ancient viewing distance Excel spreadsheet I downloaded from AVS Forums about a decade ago. I simply can't seem to find a link to it anymore, though there are alternatives online.
That is an incredible resolution, but one company actually claims to have one working right now.
That is Varjo's Bionic Display. It clearly demonstrates the screen door effect and what it takes to overcome. Though exact specifications and details are still missing, an educated guess places this display at 8192x8192 resolution per eye - close enough to our 9K target.
That said, this is almost certainly going to be prohibitively expensive for a long time to come. Varjo haven't mentioned a price yet - for good reason. It's probably going to release next year for thousands of dollars, like Microsoft's HoloLens. It's the second generation that will be the mainstream product. My guess would be 2020 before such displays make it to mainstream $300-$500 headsets.
For the next-generation of headsets due 2018/19, we should expect 4K per eye. (There are a couple of Chinese companies with 4K per eye VR headsets out this year, though they have other compromises like 60 Hz refresh rate or LCD tech.) A great step forward, but still not good enough for VR.
GPU performance. This one is the worst news of all. For the perfect VR experience, we know we need two 8192x8192 displays - this makes for an incredible 16Kx8K. Xbox One X is the first console to do 4Kx2K - and struggles to get there at 30 fps. Do note that scaling is never linear. However, we are looking at a 50x increase in GPU performance here.
What about PC? Titan V just released, and it seems to do 100-120 fps at 4K across several games. Even so, even from Titan V, we are looking at a good 15x increase. Mind you - Titan V carries an enormous ~800 mm2 die, and costs $3,000. For mainstream / console feasibility, we need a ~200 mm2 die.
How long will that take? The next-generation process technology is 7nm, coming in 2019/20, which might bring Titan V performance to performance ~$500 GPU market. Yes, we're a long way off here!
My best guess, we are probably looking at a GPU four generations out, manufactured at a 3nm EUV process. It's anyone's guess when that happens, but probably not before 2025. So, that's around PlayStation 6 / Xbox 5 time.
Unfortunately, it's gets worse. Those requirements are for games with today's graphics. So there'll always be a massive generational disparity in graphics between a game played on a TV or monitor, and on a VR headset. I suppose this is why many of the VR games this gen use cartoonish PS3/X360-era graphics.
Note that cheats like checkerboard rendering or temporal reconstruction- which are passable on a TV - will not work for VR.
All that said, there are cameras that can already shoot 8K per eye. For a 360-degree rig, several of these cameras are used, so it'll be more than enough for a final 16Kx8K experience. At least we'll have high quality videos - high enough that it looks almost real - even if the gaming experience is not up to par.
Eye tracking and foveated rendering. There's one glimmer of hope on the GPU performance front. Our eyes may seem to have a wide field of view, but really, it's only a narrow area right in front which has greatest visual acuity. So, a possible solution would be to track our eyeballs, and only render high resolution in a limited area; while the peripheral vision can be blurred and low resolution.
Current headsets have no eye tracking, and it is indeed a complex matter. There are many questions - what exactly is the falloff for peripheral vision? What happens when head and eyes move together, or fall out of sync? Will eye tracking make VR headsets more comfortable? Research is on, and we'll need to wait and see. It's hard to say how much performance uplift we can expect from this measure, but it might mean the disparities between TV/PC games and VR graphics are closed significantly.
Finally, there's the looming question of whether VR will ever be comfortable for mass consumption of games and movies? I'm afraid the answer to that is probably no. We'll probably see a fad in the middle of the next decade where everything is VR-ready, but I don't see everything being suitable for VR. My guess is I'll continue to use larger displays, complemented by VR for specific experiences. Until we get retinal implants, anyway...
