Progressive Enhancement for the VR Web

Modern VR developers are wizards of the platform. Andrew Mo commented that they are the pioneers, the ones who survived dysentery on the Oregon Trail. It was meant as a joke, but every well timed joke carries more weight when it reflects a bit of the reality and gravity of the situation. Modern VR developers really are thriving in the ecosystem against all odds in tinier markets than their Mobile app and gaming counterparts while meeting a performance curve that requires deep knowledge of every limitation in the computing platform.

John Carmack, in his keynote, said that the challenge present in Mobile VR development is like dropping everyone a level. The AAA developers become indie devs, the indie devs are hobbyists and the hobbyists have just failed.

If VR is dominated by these early pioneers then where does the web fit in? These VR pioneering teams aren't web engineers. They don't know JavaScript. While WebGL seems familiar due to years of working with OpenGL, the language, performance and build/packaging/deployment characteristics are all quite different from that of a VR application in the store. Many new skills have to be employed to be successful in the VR Web.

There is a shining light in the dark here. Most people when they hear about WebVR immediately think about games or experiences coded entirely in web technologies, in JavaScript and WebGL. It’s a natural tendency to think of the final form a technology will take or even just draw parallels with what VR is today. Since today, VR is dominated by completely 3D immersive experiences, always inside of a headset, it can be hard to imagine another, smaller step that we could take.

Let's start imagining. What does a smaller step look like? How do we progressively evolve and enhance the web rather than assuming that we have to take the same major leaps that the VR pioneers have made to date? How do we reduce our risk and increase the chance of reward? How do we increase our target market size so that it greatly exceeds the constraint of people with consumer grade VR already available?

VR Augmentations and Metadata

Our first goal has to be that existing websites continue to serve billions of users. We need to progressively update sites to have some VR functionality, but not make that a requirement for interaction. Just like a favicon can be used by a site to slightly customize the browsing experience and make their site stand out in history, favorites or the bookmark bar, a VR ready site could supply a model, photosphere, 360 video or even an entire scene. This extra content would be hidden from the vast majority of users, but would be sniffed out by the next generation of VR ready browsers and then used to improve the user experience.

One of the most compelling options is to have your website provide an environment that can be rendered around your page. In this way you can set up a simple scene, decide where your content would get projected to and the browser would handle the rest through open web standards such as glTF. This isn't even a stretch of the imagination as a recent partnership between OTOY and Samsung is working on the right standards. I was able to sync up with Jules at OC3 and I have to say, there is a pretty big future in this technology and I'm happy to add it to the list of simple things existing website developers can do without having to spend years learning and working in VR and 3D graphics. Stick a meta or link tag in your head, or push it down as an http header (this is why meta+http-equiv is probably the best approach here) and you'll get more mileage out of users with a VR headset.

It doesn't stop here though. While this changes the environment your page runs in, it doesn't allow you to really customize the iconography of your site the way a simple, 3D model would be able to. Another glTF opportunity is in delivering favicon models that a browser can use to represent sites in collections like the tab list, most recently visited sites and favorites. A beautifully rendered and potentially animated 3D model could go a long way to putting your website on the mantle of everyone's future VR home space.

I think there is more value to be had in the Web Application Manifest specification too. For instance, why can't we specify all of our screenshots, videos and icons for an installable store page? A VR Browser would now be able to offer an install capability for your website that looks beautiful and rivals any existing app-store. Or if you like the app-store then you can specify the correct linkage and "prefer" your native experience. The browser in this case would redirect to the platform store, install your native application and off you go. I see this as exceptionally valuable for an existing VR centric developer who wants to augment their discovery through the web.

Immersive Hybrid Experiences

Our next goal is to start incrementally nudging our users into the VR space. While augmentations work for existing VR users and are ways to provide more VR like experiences on the existing web, we can do even better. Instead of a VR only experience we can build hybrid, WebGL+WebVR applications that are available to anyone with a modern browser.

How does this work? Well, we start with the commonality. Everyone has a browser today capable of some basic WebGL. This means any experiences you build can be presented to that user in 3D through the canvas tag in a kind of magic window. You can even go full screen and deliver something pretty immersive.

To improve this further, we can abstract input libraries that work across a large set of devices. Touchpad, touch, mouse, keyboard, device orientation, gamepad and the list continues to grow each day. By having a unified model for handling each type of input you can have a great experience that the user can navigate with mouse and keyboard or spin around in their chair and see through the window of their mobile phone. Many of these experiences start bordering on the realism and power of VR without yet delivering VR.

The last thing we do to nail this hybrid experience is detect the presence of WebVR. We've landed a property in the spec on the navigator object, vrEnabled. This will return true if we think there is a chance the user could use the device in VR. While I think there are some usability issues with such a simple property that will result in maybe some browser UX for turning VR on and off, this is a great start.

This is the next level of write once, run anywhere, but its built on the concept of progressive enhancement. Don't limit your user reach, instead scale your experience up as you detect the capabilities required. I've recently stated that this is a fundamental belief to our design of WebVR and I truly believe in maintaining this approach as long as there are users who can benefit from these choices.

I wanted to give an example of one of these experiences and so here you can see a 360 tour we've linked from our Oculus WebVR Developer portal. There will be many more examples to come, but this will run in any browser and progressively enable features as it detects them. My favorite is simply going there in your phone and looking around using device motion.

I can't speak highly enough of the value in building experiences like this. For this reason we at Oculus and Facebook will be releasing numerous options for building these experiences in minutes to hours rather than in days or more. Both content generation and viewing needs to be made easier. We need to make great photosphere export from pretty much any application (any developer should be able to emit and share a photosphere from anywhere in their game/application/experience), optimize how we transmit and display photospheres with potential improvements to streaming them in, with simple to use libraries. It has to be just as easy to extend this to use 360 video. Even a simple looping 360 video could provide your user with AMAZING bits of animation and value. Finally, extending this to make it interactive with text and buttons. You can go further if you have the skills to do so, but the basic libraries such as React VR will allow anyone to create great experiences with the above characteristics.

Getting out of the Way

Once we've extended existing websites, gotten some simple libraries in place and people start to build VR content, then final step is get the hell out of the way. This is easier said than done. There is a large contingent of web technology naysayers that have pointed out certain flaws in the web platform that make it non ideal for VR. I'll give some examples so it is clear that I do listen and try to resolve those issues.

1. JavaScript is too slow. To which I respond, maybe when it comes to new experiences. JavaScript is a tuned language. It is tuned by browser vendors based on the content out there on the web today. Tomorrow, they could tune it differently. There are so many knobs in the run-time. C++ is no different. There are knobs you can tune when you pipe your code into cl.exe and it WILL result in differences in the run-time behavior and speed of your application. The browser is running in a chosen configuration to maximize the potential for a set of sites determined by usage.
2. GC sucks. To which I reply, yes, the general model of GC that is present in your average web runtime does indeed suck. However, this is again, only one such configuration of a GC. Most GC's are heuristic in nature and so we can tune those heuristics to work amazingly well for VR or hybrid scenarios. I won't go into details, but let's just say I have a list of work ready for when I hire my GC expert into the Carmel team ;-)
3. Binding overhead is too high. To which I respond, "Why?" There is no requirement that there be any overhead. An enlightened JIT with appropriate binding meta-data can do quite a bit here. While there are some restrictions in WebGL binding that slow things down by-design I have high hopes that we can arrive at some solutions to fix even that.

That list is not exhaustive. When you have full control over the environment and aren't part of a multi-tenant, collaborative runtime like a web browser, then you can be insanely clever. But insanely clever is only useful for the top 1% of applications once we enable anyone with a browser to build for VR. We need to get out of the way and make the common cases for the remaining 99% of the content not suck. That’s our space, that's our opportunity, that’s our goal.

Beyond the standard arguments against VR and the web I think there are more practical issues staring us in the face. The biggest one is the prevalence of full VR browsers. The first fully featured VR browser is going to be a deep integration of the web platform, OS, device and shell. There is simply too much going on today for a web application to be able to run with zero hiccups and an overall lack of measurement and feedback between the various sub-systems. Tuning everything from the async event loop, thread priorities and which sub-systems are running when in and out of VR is a very important set of next steps. Combining these enhancements with UX and Input models that enable people to have a level of trust in their to the extent that they could interact with a merchant in virtual space, whatever form that might take.

Right now we are experiencing the power of VR shells that plug into basic browser navigation models and do simple content analysis/extraction. This is an early days approach that only achieves a small fraction of the final vision.

For the graphics stack we need some basics like HTML to Texture support in order to bring in the full power of the 2D web into our 3D environments. I've been referring to 2D UX as having a "density of information" that can't be easily achieved in 3D environments. We need this in order for VR experiences to enable deep productivity and learning. Think about how often, in our "real" 3D world you break out your phone, laptop, book and other 2D format materials to quickly digest information. I don't think VR is any different in this regard. John Carmack, during his keynote, also noted that there was huge value in 2D UX because we've had so many years of experience developing. I believe this to be true and think that an HTML to Texture will broaden the use cases for WebVR dramatically.

We also need to enable full access to the pipeline. More and more I'm hearing about the necessity for compute shaders and access to gl extensions like multi-view. Even native is insufficient for delivering super high quality VR leading to advances in hardware, OS and software. The web needs to provide access to these capabilities quickly and safely. This may mean restrictions on how we compose and host content, but we need to experiment and find those restrictions now rather than holding the web back as a core VR graphics platform.

To close out, note how this section details yet more progressive enhancement. Each of the new capabilities and improvements we make will still take time to filter out to the ecosystem. This is no different in the native world where extensions like multi-view which have been defined for a couple of years are still not uniformly distributed. So producing experiences that reach the largest possible audience and gradually increase capability based on the device and browser features you detect is going to be key.

Over the next few months I'll be providing some tutorials talking about strategies and libraries you can use to enable VR progressive enhancement for your existing sites. You can also check out more of our demos and sign up for more information about our libraries like React VR at the Oculus Web VR Developer Portal.