Transcend 3D Window Switcher - TreeHacks

A few weekends ago I went to the TreeHacks hackathon at Stanford. It was a lot of fun, and was probably the best hackathon I've been to so far, so good job Stanford people! While there, I got a chance to work with a Meta 1 developer kit. I used them to make a 3D augmented reality replacement for alt-tab, where the windows on your computer would fly out into 3D space.

For those of you who don't know, the Meta 1 augmented reality glasses are a pair of glasses that you can wear that allow you to render objects on top of the world in real time. You can reach out and grab objects and interact with them, too! It isn't quite perfect yet, but the developers are hard at work; the day before I got there, they finished the first version of a new positional tracking system they were working on.

Anyway, my idea was to make it so that you could use the extended screen space offered by augmented reality to free your computer windows from the screen. I thought it would be a lot more fun and more intuitive if you could simply grab and drop windows in 3D space, with your hands. Here's a youtube video of my finished product:

My hack is split into two main parts:

• A C++ Unity plugin that uses the Win32 APIs to identify the open windows and then take a screenshot of each window. It also loads those window screenshots into a Unity texture.
• Some C# Unity scripts that create a quadrilateral object for each window, create a texture to hold the image, and control the window object animations and interactions.

This hack was challenging because C++ and C# are languages I am only somewhat familiar with, and I had never worked with the Win32 APIs or Unity scripting before. As such, I ran into a few fun bugs. Some of my favorites:

• In C#, most things are passed by reference, like in Java, but if you declare something as a "struct" instead of a "class", it becomes pass-by-value. This resulted in some very strange scaling behavior as I manipulated my windows, as the windows kept their scaling but the object I was using to control the smooth transitions did not change with it.
• My project would inexplicably crash whenever I tried to test it as an application, but as soon as I tried to debug the problem, it went away. This turned out to be caused by an extremely nonintuitive problem:
  1. When you minimize a window in Windows 8, usually it keeps its width and height. However, when you minimize a window using the "minimize all to desktop" button, all of the windows get widths and heights of zero.
  2. When you try to make a texture of zero size, Direct3D gives you a null pointer instead of actually making a texture.
  3. When you try to copy a zero-size image into a texture that is actually a null pointer, your program crashes. It turned out that whenever I tested it as an application, I was using the minimize button to make it easier to see, and thus every time I did that, the problem would occur and it would crash.

Overall, though, it was a very positive experience. I talked to a bunch of people from Meta, and they liked my project. I actually ended up winning one of their Meta 1 Developer Kits to take home!