G4′s Attack of the Show! ambushed our friends and neighbors at the Microsoft campus, capturing a whirlwind of video highlights and a cameo  of Vectorform’s Galactic Alliance HD. The four-player tower defense game made a big splash at CES this year, where it could be experienced on the new Samsung SUR40 for Microsoft Surface. Check out the demo from Director of Research Steven Bathiche, at 3:43.

Also showing up for Jessica Chobot’s campus tour was Skinput, a wickedly-clever project from Microsoft Research that turns the body into an input surface; Kinect Fun Labs, a new addition to the Kinect Hub; and our buddy Steve Clayton, a Microsoft storyteller who has covered our work with Kayak and Virtual Colonoscopy.

Maybe we should invite the gang over to see our latest? We do get the most surprising visitors…

Vectorform’s multi-touch user interface allows physicians to view and manipulate a 3D rendering of a patient’s colon. Using software to translate a series of CT scans into a 3D representation, the VC Viewer provides physicians with life-like imagery and seamless navigation using natural gestures including pan, zoom, pinch, traverse and rotate. This Natural User Interface (NUI) solution was highlighted at TechFest 2011, which brings Microsoft researchers and Microsoft product teams together in Redmond, WA for an event where attendees see the innovative technologies emerging from Microsoft’s research efforts.

Read a tech-centric overview of NUI and Microsoft Research over at Softpedia, or learn more about the VC Viewer with Marianne Kolbasuk McGee’s article on InformationWeek, where Hiro Yoshida, Ph.D, Director of 3D Imaging Research, Dept Radiology, Massachusetts General Hospital highlights the solution. The interview details how the process aids in reducing the processing of the laxative-free virtual colonoscopy scans to three minutes, and how it helps healthcare providers respond more efficiently if a polyp is found.

We found a great summary of why it’s important to talk tech and talk about the VC viewer on Next at Microsoft. “Today this is an invasive process that scares many people away from what can be life saving procedure as the idea of sending a tube with a camera in to your body isn’t especially appealing…This had led to a virtual colonoscopies (using a CT scanner) that now take about 3 minutes instead of 30+ minutes…There is a lot of other technology at play here – our high performance computing (HPC) platform, Microsoft .Net 4.0, and the Intel Parallel Studio 2011 developer tool as well as a fully parallelized GPU-based volume rendering engine developed by Microsoft Research. I know…that sounds like a mouthful but when it can help save lives and reduce invasive procedures, it’s worth it,” Steve Clayton writes.

See Principle Architect at Microsoft Platform Developer Evangelism Curtis Devlin demonstrate the VC Viewer at TechFest below.

Check it out, and while there, we also recommend B2C’s coverage of Foursquare’s growth from six to seven million users, all between January and February of this year. With Foursquare Playground officially open for business, it will be interesting to see what March and April bring.

As for our work with Foursquare, Microsoft and Fiat/Impatto, it is a busy and exciting time. Our next SXSW spotlight hits on a Fiat brand experience, and the continued efforts of the Vectorform squad in Austin, Texas, so stay tuned here.

How do we go about re-inventing and redefining the technical parameters of online experience? HTML5, CSS3, and pure UX magic. Hear master magician Dominic Espinosa talking about the AP Timeline Reader below, or wander over to Renee DeCoskey’s detailed overview at Business 2 Community. News reading reinvented, and it’s got pictures!

Microsoft Showcase: Developer Interview With Vectorform

There are of course multiple ways to deliver a stereoscopic experience; linear and circular polarised glasses paired with filtered projection (IMAX and Disney RealD), lenticular or masked parallax displays (such as Sharp 3D or the Nintendo 3Ds), and many more, including the easiest and oldest — anaglyphic. While I’ll discuss anaglyphic compositing in some upcoming articles, this tutorial covers some of the actual camera setups and rendering tricks needed to create stereoscopic imagery in the first place. Generating content for stereoscopy (left and right sides) is universal, regardless of delivery mechanism, so this tutorial should be suitable for any system you’re working with.

Two schools of thought

When shooting Avatar, James Cameron used converging cameras. This is where the left and right cameras are mounted side by side, but rotated towards each other to aim at a specific z-axis location. Advantages include immediate directorial control over where the image lies on screen; the apparent “base” of the stereoscopic effect, where an item is balanced between jumping out of the screen at the viewer, and receding into the distance. The disadvantage is large amounts of perspective and barrel distortion, which must be fixed in post before being usable on screen.

Pixar took the opposite approach in Up – using (in their case, virtual) cameras mounted exactly parallel to each other. This results in precise perspective and no distortion, albeit with the need for post render adjustments to place the stereoscopic base (though I’m not actually sure how Pixar handle this, they may have done it in-render as well). This was paired with floating screen frames and other techniques to precisely tune the 3D effect seen in theatres.

Lightwave camera rig

While Lightwave offers a built-in stereoscopic camera, it doesn’t support features like photo real motion blur, nor does it let us tweak the details of the stereo base or convergence. The good news is that we can build a far more advanced stereo camera rig using cameras, parenting, and expressions.

To start off with lets create a Camera Main, and attach it to a null. Every artist has his or her preferred camera rig; this is the basic one I use for most setups, even if it’s just a 3/4 shot of a digital illustration. Creating a main camera also allows composition and animation from a neutral point, instead of favouring one eye over the other.

Stereo cameras

Clone the main camera twice, and rename as Camera Right and Camera Left. Parent them to the main camera, and shift Camera Right on the X axis. Depending on your scene scale and desired depth effect, this could be anywhere between 10mm and 1m, or more.

Select Camera Left and open the Motion panel. Add Follower, select Camera Right, and set all but the position channels to none. Change the X axis multiplier to -1 and click Continue to apply.

You should now have two cameras on either side of your main camera, perfectly mirroring each other.

Hybrid Convergent-Parallel system

This is where the magic starts to happen!

Add a second null, name it Focal Null, and parent it to Camera Main as well. Set X and Y axis to 0, and uncheck them by clicking on the axis label. No need for unnecessary axis handles cluttering up the viewports.

If you try aiming the left and right cameras at this null, you’ll notice severe distortion when viewed in stereo, but leaving the cameras parallel will leave you without control over depth convergence. This is why we want a hybrid system, and it’s also why Lightwave is perfect for the job!

Select Camera Right and change the camera type from Perspective Camera to Shift Camera. You’ll see a new window pop up with extended options. Uncheck use camera pitch and click the e button next to horizontal shift, then do the same for Camera Left.

In the opened Graph editor, switch to the Expressions tab, and click New. Enter Camera Left as the name, and the following expression as the value:

([Camera Right.Position.X]/[Focal Null.Position.Z])*0.9

Click Apply to attach it to the Camera Left Horizontal Shift property, then switch back to Camera Right and repeat with a new expression, using this string:

([Camera Right.Position.X]/[Focal Null.Position.Z])*-0.9

If you select both cameras, you should see the viewport lines intersecting with each other at the same z-depth as your focal null. Depending on the rendered resolution, you may need to adjust the multiplier at the end of the expression to achieve this. 0.9 is the magic number for HD resolutions (16×9 ratio), but you can also add a few expressions to do this dynamically, along with adding support for animated zooms.

A fully dynamic rig, however, requires the use of LScript format code to access non-animated variables, which doesn’t support objects containing spaces. Any item names with spaces will have to be renamed without them.

Create an expression named “Zoom” and attach it to the focal length channel of both left and right cameras, using the following code:

CameraMain.zoomfactor(Time)

Create two more expressions titled “Aspet” and “Position”, using the following lines:

Scene.width/Scene.height

[CameraRight.Position.X]/[FocalNull.Position.Z]

This calculates both the distance scaler and the rendered aspect ratio. Using these as sub-expressions, we can simplify the left and right expressions like this:

[Position]*([Zoom]/[Aspect]*0.5)

[Position]*([Zoom]/[Aspect]*-0.5)

This convergence of the camera views indicates the effective zero point of the depth effect – the point at which objects will appear to be at the same level as the screen. It’s the best of both worlds, giving full control over stereo convergence while avoiding the perspective distortion of typical converging cameras. Careful use of this setup can result in 3D renders needing very little adjustment in post, with easily customised, even animated, depth controls in Lightwave.

Download

Download Lightwave scene file with basic camera rig (standard HD setup).

Download Lightwave scene file with dynamic camera rig (calculates focal length and adds dynamic support for any resolution / ratio selected).

The Art of Stereo

There are countless nuances when it comes to rendering for stereoscopic presentation, most of which I’m still learning myself. As a start though, here are a few basics…

Poking the viewer in the eye is not a classy thing to do. Keep it subtle, logical, and always subservient to the story. Learn a lesson from Pixar; if the 3D effect is ever distracting or detracting from the emotional tone, pull the left and right cameras closer together to flatten out the scene. If you get the chance to watch Up again in 3D, keep an eye on the visual depth of the scene. It’s almost always inversely proportional to the emotional depth.

Keep elements within an easily focusable area. Too much difference between between foreground and background elements, and it’s painful to watch. Too little, and there’s no reason to see it in 3D. Set the distance between left and right cameras (the stereo base) to a width that balances between a realistic, pleasant, and appropriately dramatic feeling of depth. Filmmakers will of course experiment with depth like Hitchcock experimented with the dolly-zoom combination; creatively changing the perspective and depth within a scene to achieve dizzying effects. If you are that filmmaker, please don’t overdo it!

Blurry or nebulous objects can destroy depth. Not necessarily a bad thing in all circumstances; Avatar used plenty of depth of field in some shots, and scenes still felt nicely dimensional. But blurry items, and motion blur in particular, make it much harder for the brain to process apparent depth. Most stereoscopic displays are already taxing the visual system somewhat (especially shutter and anaglyphic glasses); there’s no need to make it even harder. While you may have a 180º shutter on your main camera (50% motion blur in Lightwave), the left and right cameras may need to use half that, if not less, depending on the movement within the scene and the style requested by the director(s).

Like pretty much any other artistic endeavour, we’re here to manipulate and even outright trick the brain. Keep it sane, painless, and most of all, fun!

Compositing and Delivery

Finishing depends entirely upon the delivery mechanism; analgyphic (anachrome, triochrome, colorcode), polarisation (IMAX linear, RealD circular), lenticular (Sharp 3D), and many others.

Newtek and Lightwave 3D are either registered trademarks or trademarks of Newtek Incorporated in the United States and/or other countries.

Adobe, Photoshop, and After Effects are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Would you like to learn more? Check out Microsoft’s Curt Devlin’s blog post about the application and the demonstration in Boston. 

Virtual Colonoscopy will also be demonstrated at the RSNA Conference in November, stay tuned for more updates!

This is how the paper cutout effects were designed and generated dynamically using Adobe After Effects.

Animation with alpha

For our example video, character animation was handled in After Effects. Having broken down the script and subsequent voice acting into paragraphs, the lipsync was animated in Papagayo (see Lipsyncing with Papagayo for more on this important aspect of the high-speed workflow) and the elements were animated in individual compositions for each clip.

In other cases, the animation may be a 3D rendering, from another application, or hand drawn. The biggest concern here is that the elements must have full alpha, with no backgrounds or half-transparent portions (antialiasing is good, but there may be issues with large patches of partial transparency once we start playing with alpha boundaries).

Paper textures

Depending on the desired effect, this can take all sorts of forms. For the Microsoft PowerPivot video, a sequence of crumpled paper photos were used along with frame blending in After Effects. This was slowed down (again, using frame blending) till the desired look was found. The final paper texture animation should be the same size and length as the other comps.

Using directional R/G/B lighting when photographing the crumpled paper could make it possible to develop false geometric distortions in After Effects, gaining a little more realism in how the imagery follows the shape of the paper. This tutorial, however, only covers the use of plain-lit paper textures.

Effects

The animation and paper compositions should be added to a new comp of the same size, and an adjustment layer and plain white solid should be created.

adjustment layer

Two expression slider controls allow us to globally control the strength of the displacement effects (to be applied to the animation and white solid layers), while the drop shadow performs a very basic role; add a little bit of lighting to an otherwise flat paper effect.

animation

Because After Effects’ Displacement Map effect is unidirectional, two are applied to help give a little more depth and variation. The displacement source for each effect should be set to the paper texture layer. Option-clicking on the animation badge (the stopwatch icon) lets us add expressions to the Max Horizontal and Max Vertical displacement values.

For the first Displacement Map effect, apply this expression to both Max Horizontal and Max Vertical:

thisComp.layer("Adjustment Layer 1").effect("Displacement Slider")("Slider")

The second Displacement Map effect needs this expression applied to both Max Horizontal and Max Vertical:

thisComp.layer("Adjustment Layer 1").effect("Displacement Slider 2")("Slider")

Lastly, a Curves adjustment is added to help the colours and values pop. Adding all the shading, textures, and distortions can start to muddy and dull the animation, so curves are used throughout to brighten and enhance the imagery.

Set the layer Transparency Mode to Multiply to apply it to the underlying layers.

white solid

This is the most complicated of the layers, with two main purposes; to expand and create the torn paper edges, and add depth to the underlying paper textures.

The Set Channels effect is important, as we use this to grab the alpha channel from the animation layer. Select layer 2 in the source layer 4 input, all the other settings can be left at their defaults.

Use Simple Choker to expand the alpha, creating the extra white space needed for the torn paper edge.

Just like the animation layer, add two Displacement Map layers with inputs set to the paper texture, and the following expressions applied to Max Horizontal and Max Vertical displacement values.

thisComp.layer("Adjustment Layer 1").effect("Displacement Slider")("Slider")

thisComp.layer("Adjustment Layer 1").effect("Displacement Slider 2")("Slider")

To finish the tearing of the paper edge, apply a Turbulent Displace effect and tune to taste. The amount, size, and complexity will all depend on the scale of the compositions and the desired style. Because the previous displacements are based on the animated paper, and the turbulence is based off of the new alpha shape, the effect will change automatically based on the strength and speed of the paper animation. Animating the evolution is possible, but typically adds far too much action and confusion.

After Effects’ Emboss is rudimentary at best, but once applied, we can add Channel Blur to soften the results into a usable effect (taking care not to blur the alpha). Matching the emboss relief value with the R/G/B channel blurs tends to work best. The direction is left at 0 degrees, giving a slightly concave appearance. This helps create the appearance of a slight page curl along the edges.

Lastly, a Curves adjustment is added to brighten up the output and improve the final composited tonality. Setting the layer transparency mode to Soft Light allows the emboss effect to adjust the underlying paper texture.

paper texture

Set the Track Matte to use the above layer’s Alpha, and the look is almost complete. In the example project, Levels were used to brighten the paper texture. This depends, of course, on the paper textures used and how dark the resulting paper crumple needs to be.

[media id=1 width=500 height=281]

Final Compositing

Once completed, the composition is ready to be used in the final edit. For graphics that border the edge of the composition, there may be some artifacts left over from the multiple displacements. To fix this, either increase the size of the pre-comps to slightly larger than your output, or scale up the comp in the final video to cut off edges.
[media id=2 width=500 height=281]
Watch the final PowerPivot video on YouTube or Vimeo to rate it or leave a comment!