Using Cameras in AR

One more object we haven't discussed yet that is essential for any Unity application is the Camera. Unity developers may often add a Main Camera to their project scene and not give it much more thought. But in AR, the camera is especially important.

Cameras are devices that capture and display the virtual world for the player. For rendering computer graphics, the camera's Pose (position and rotation transform), its rectangular Viewport and Field of View (FOV) together define how much of the scene is visible and rendered on the screen. The camera Pose is where the camera is pointing. The Viewport is like a rectangular window that we're looking through; anything outside the Viewport is clipped and not drawn. The Field of View defines the viewing angle. In regular video games, developers enjoy the option to modify any of these options to give the cinematic camera effects they want to emulate on the screen.

But for augmented reality, the constraints on these parameters are dictated by the physical device running the app. The perspective view must match that of the real world, as seen through the user's device video camera or their own eyes' view through wearable glasses (for video see-through or optical see-through AR devices, respectively). The Viewport will correspond to the device's camera view or FOV of eyewear.

Fortunately, most of the default settings of the Camera object or camera prefabs provided with your AR SDK are good. The following screenshot shows the Inspector pane of a Camera with the Vuforia prefab:

In Unity, when a new frame is rendered in the frame buffer memory, it is cleared to a skybox image (like a nice blue sky with clouds or a 360-degree image), or a solid color. For AR, we want the background to be transparent and with the only objects rendered on each frame being those that are visible, without any background skybox or anything else. Specifically, in our AR view, any pixel in the frame buffer without a color (that is, black) will be transparent. On optical see-through devices, you'll look through the glasses when there are no colored pixels. On video see-through devices, the camera view frame will be visible as the background. Therefore, in our Camera, we set the Clear Flags to Solid Color, and the Background color to black (0, 0, 0).

The following Hierarchy shows the Vuforia AR Camera prefab. It contains a Camera as already described. Then there's a BackgroundPlane that is a child of the Camera. The video image is painted on this plane when the scene is rendered:

AR cameras for AR SDK also have special scripts, or Components, attached that implement the interface with the underlying toolkit SDK algorithms and device drivers. The specifics vary from one toolkit to the next. Sometimes you'll be instructed to review and modify the parameters in these components to configure the behavior of your AR application.

In newer versions of Unity, AR cameras are being introduced with native support in Unity. At the time of writing this, the only built-in AR camera is for Microsoft HoloLens. In this case, you may not need any special AR components on the Camera object. Instead, we'll use Player Settings to enable Virtual Reality Supported and choose the Windows mixed-reality holographic SDK. We will explain these steps in detail in the next chapter. Stay posted for built-in support for other AR devices as they become available directly from Unity.

Although Unity presently provides built-in support for HoloLens, at the time of writing this, we cannot preview Play mode without a HoloLens device connected or the emulator installed. Similarly, the Apple ARKit offers a remote control feature for Play mode when an iOS device is connected. Other SDKs (Vuforia and ARToolkit) do support Play mode previews in the Game panel of the editor with only a standard webcam attached to your PC, as we'll see in the next topics. Therefore, in most cases, your PC should have a working webcam attached. On Windows, you can see it in the Windows Start Menu | Device Manager | Imaging devices | your device name.

In the rest of this chapter we help you add an AR SDK to your Unity projects. We consider two general toolkits, which both have Unity interface packages: Vuforia and ARToolkit.