In a previous article I described how to display full-spherical stereoscopic images and videos in Unity. In this article I’m going to start looking at how to create that content.
One approach (described in detail on the excellent eleVR site) uses a rig consisting of multiple cameras to capture stereoscopic image pairs which are then stitched together in software to create a pair of spherical images, one for the left eye and one for the right. The results can be quite good, but there are two fundamental problems with this approach — one technical, one practical.
The technical problem has to do with the need for the cameras to have overlapping fields of view in order for the stitching software to work. That inevitably means that there’s a minimum distance between the center of the camera rig and the objects in the scene. Anything closer than that won’t be visible, and certainly won’t be seen stereoscopically. That distance is typically about four or five feet, which is unfortunate since that’s the distance range in which stereoscopic depth perception is the strongest.
The practical problem is that cameras are expensive. I have yet to see a spherical stereoscopic rig with fewer than six cameras with very wide-angle lenses, so you’re looking at several thousand dollars for a full rig. That’s a lot of money, certainly for a hobbyist, and definitely for me.
Fortunately, capturing images from the real world is not the only way of generating spherical stereoscopic content. You can use CGI to generate the images entirely in software, so there are no hardware costs and no limitations on how close objects can get to your (virtual) camera. I suspect that most of the immersive film style content that will be available in the next few years will come from large animation houses like Pixar rather than from people shooting actual video footage.
How it Works
The basic idea is to create the software equivalent of a stereoscopic camera rig, with a pair of virtual cameras separated by some distance (typically, something close to a normal inter-pupillary distance (IPD) of 60 millimeters or 0.06 meters). For each camera, we render a series of 360 narrow vertical strips, each covering 1 degree of longitude and 180 degrees of latitude. We rotate the cameras about their common pivot point by 1 degree for each strip. The virtual cameras use an equirectangular projection, so each vertical strip occupies the same width across our final output image.
Once we have all these image strips, we then join them together to form a pair of images (left eye and right eye). Finally, we combine the two images into one frame using an over-under layout, and we’re done.
If you’re interested in a more detailed description of all this, you should definitely take a look at this paper that Paul Bourke wrote over a decade ago. It describes it all really well, with lots of pictures. 🙂
The question is, what software should we use to do the rendering? There are several options, but most of them cost more than the stereoscopic camera rig would! If you have a budget for your project, you could use Maya or 3D Studio Max along with the free Domemaster3D shader, which appears to do exactly what you need.
However, we want to be able to do all this using entirely free software. There’s Blender, of course, but it does not seem to know how to render a spherical image (at least, not as of January 2015 — hopefully that will change).
For a while, I scratched my head. There is a lot of commercial rendering software that can do it, but nothing free and/or open source. Until I remembered an old friend…
Enter POV-RAY
Back before I became involved in VR (a long, long time ago), I was interested in pre-rendered computer graphics. One of the cleverest ways of creating beautiful images was a technique called raytracing. A raytracer works by firing an imaginary beam from your virtual camera position through a particular pixel in the output image, and seeing what it hits in the scene. For every point it hits, you generate additional rays representing reflections and refractions and continue tracing those paths until you reach a light source. You then repeat that for every pixel.
Back in the day, there were lots of raytracers out there. One reason for that is that they’re easy and fun to write, and a lot of people create them for school assignments. I wrote one myself in C, and it was only a few pages long. Of all the raytracers I tried (and I did a ton of side-by-side comparisons), there was one that stood out above all the others — the Persistence Of Vision Raytracer, or POV-RAY.
Flash forward quite a few years, to the present. When I started looking for free software-based renderers, one of the first ones I came across was my childhood friend POV-RAY. It brought back some nice memories. Out of curiosity, I looked for some of his contemporaries. Sadly, they had all passed away leaving nothing but 404 errors. Fortunately, POV-RAY is just what I needed.
POV-RAY, like most raytracers, uses a scene description language. You create a text file containing a description of all the objects in the scene, all the light sources, and the position and properties of your virtual camera. Then you run the raytracer on that file, and a little while later you get a beautiful image of your scene. Depending on the complexity of your scene, “a little while later” can vary quite a bit. A simple scene might render in a few minutes. A more complex one would have to run overnight. A really complex one… well, you may want to go on a nice vacation to someplace warm and sunny while it runs.
The good news is that raytracing parallelizes really well, so if you have (for example) 360 computers, you can render an entire image in the time it takes one computer to do a single strip. And modern raytracers can take advantage of multi-core CPUs (or better yet, GPUs) to speed things up dramatically.
Anyway, all we have to do is create our scene description and let POV-RAY do the rest.
Input Files
POV-RAY uses two different kinds of input files. The scene description itself is stored in a file that (typically) has a “.pov” extension. The information about the rendering process (resolution, output file format, things like that) are stored in a configuration file that has a “.ini” extension. Basically, the .pov file says what to render and the .ini file says how to render it.
We’re going to create two .ini files, one for the left eye and one for the right.
Here’s our left.ini file:
;; Configuration file for rendering the left-eye strips Input_File_Name=left.pov Output_File_Name=frames/left.tga Output_to_File=on Output_File_Type=T Antialias=on Antialias_Threshold=0.01 Display=off Verbose=off ;; each frame is 10 pixels wide (so 3600 pixels in total) ;; by 1800 pixels high Width=10 Height=1800 ;; Make the panoramic in 1 degree slices Initial_Frame=0 Final_Frame=359 Initial_Clock=0 Final_Clock=359
This says that the file to render is “left.pov” and the output files should be stored in the “frames” directory and should be named “left000.tga”, “left001.tga” and so on (TGA is an old-fashioned image file format). We render 360 frames in total, numbered zero through 359.
The right.ini file is very similar. 🙂
Now we need to set up the .pov files. Rather than duplicate the scene description, we’ll store that in a separate file and use POV-RAY’s #include system to reference it (this should be familiar to anyone who programs in C). The left.pov file is tiny, and looks like this:
#version 3.7; #declare EYE = 1; #include "commonparameters.inc" #include "stereopanoramic.inc" #include "scene.pov"
The right.pov file is again quite similar. The EYE variable is set to 1 for the left eye and 2 for the right eye. We store a couple of rendering parameters that are common to both eyes in a separate file called commonparameters.inc and #include that here.We also #include a file called stereopanoramic.inc, which sets up the virtual camera for stereoscopic rendering. That file originated on Paul Bourke’s site (see the link earlier in this article). I just made a couple of modifications to change the rendering from “perspective” to “spherical” and make the vertical field of view 180 degrees. Finally, we #include the actual scene description file called scene.pov. When creating scene.pov, remember to omit any camera since the camera gets set up in stereopanoramic.inc.
The commonparameters.inc file just has two lines:
#declare FOCAL = 0.75; #declare CENTER = <0.0, 0.5, 0.0>;
These specify the focal length and the position of the camera (which you will want to change, to put it in an appropriate position in your scene).
With all this set up, we just run the POV-RAY renderer on each of the .ini files:
“c:\Program Files\POV-Ray\v3.7\bin\pvengine64.exe” /exit left
“c:\Program Files\POV-Ray\v3.7\bin\pvengine64.exe” /exit right
The result is two sets of image strips in the “frames” directory.
Image Magick
To assemble the strips, we again turn to an ancient but powerful piece of software called Image Magick. It can do all kinds of amazing image transformations, entirely from the command line. We’re only going to use one of its features, though — a program called “montage” that can assemble images together to form mosaics. For our purposes, this will do the trick:
montage frames\left*.tga -tile 360×1 -geometry +0+0 left.jpg
montage frames\right*.tga -tile 360×1 -geometry +0+0 right.jpg
The first command takes all the left*.tga files and assembles them into one, tiling them 360 horizontally and 1 vertically with no borders (+0+0) and converting the result into a jpeg file called “left.jpg”. The second command does the same for the right image strips.
One final command will combine the left and right images into our over-under format:
montage left.jpg right.jpg -tile 1×2 -geometry +0+0 output.jpg
If you want to do a side-by-side layout instead of over-under, just replace “1×2” with “2×1”.
The resulting output.jpg file is suitable for viewing using any stereoscopic image display software, including the Unity software I described in my previous article.
You may be wondering why I’m using command-line tools for all of this, rather than something with a graphical user interface. There is in fact a piece of software out there called Slitcher which does the joining up of the strips. I’ve never used it, but I’m sure it would work just as well as Image Magick. However, the big advantage of command-line tools is that you can use them in batch files which can run unattended for long periods of time. Both POV-RAY and Image Magick can also run on Linux systems, which is great if you want to set up your own small render farm or buy time on a commercial one.
Anyway, that’s it for now. All of the code for this article, including a sample scene, can be found here.
Have fun!