# Chapter 6. Blending and Augmented Reality

All colors are the friends of their neighbors and the lovers of their opposites.

Marc Chagall

If you’ve ever used Photoshop to place yourself in front of the Taj Mahal in a paroxysm of wishful thinking, you’re probably familiar with layers and opacity. Alpha simply represents opacity on a zero-to-one scale: zero is transparent, one is fully opaque. Alpha can be used both with and without textures, and in this chapter we’ll pay special attention to textures that contain alpha. Blending is the process of compositing a source color with an existing pixel in the framebuffer.

Tangentially related to blending is anti-aliasing, or the attempt to mask “jaggies.” Antialiased vector art (such as the circle texture we generated in the previous chapter) varies the alpha along the edges of the artwork to allow it to blend into the background. Anti-aliasing is also often used for lines and triangle edges, but unfortunately the iPhone’s OpenGL implementation does not support this currently. Fret not, there are ways to get around this limitation, as you’ll see in this chapter.

Also associated with blending are heads-up displays and augmented reality. Augmented reality is the process of overlaying computer-generated imagery with real-world imagery, and the iPhone is particularly well-suited for this. We’ll wrap up the chapter by walking through a sample app that mixes OpenGL content with the iPhone’s camera interface, and we’ll use the compass and accelerometer APIs to compute the view matrix. Overlaying the environment with fine Mughal architecture will be left as an exercise to the reader.

## Blending Recipe

Some of my favorite YouTube videos belong to the Will It Blend? series. The episode featuring the pulverization of an iPhone is a perennial favorite, seconded only by the Chuck Norris episode. Alas, this chapter deals with blending of a different sort. OpenGL blending requires five ingredients:

1. Ensure your color contains alpha. If it comes from a texture, make sure the texture format contains alpha; if it comes from a vertex attribute, make sure it has all four color components.

2. Disable depth testing.

glDisable(GL_DEPTH_TEST);
3. Pay attention to the ordering of your draw calls.

4. Enable blending.

glEnable(GL_BLENDING);

glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

For step 5, I’m giving a rather classic blending equation as an example, but that’s not always what you’ll want! (More on this later.) Specifically, the previous function call sets up the following equation:

S is the source color, D is the starting destination color, and F is the final destination color. By default, OpenGL’s blending equation is this:

Since the default blending function ignores alpha, blending is effectively turned off even when you’ve enabled it with glEnable. So, always remember to set your blending function—this is a common pitfall in OpenGL programming.

Here’s the formal declaration of glBlendFunc:

void glBlendFunc (GLenum sfactor, GLenum dfactor);


The blending equation is always an operation on two scaled operands: the source color and the destination color. The template to the equation is this:

The sfactor and dfactor arguments can be any of the following:

GL_ZERO

Multiplies the operand with zero.

GL_ONE

Multiplies the operand with one.

GL_SRC_ALPHA

Multiplies the operand by the alpha component of the source color.

GL_ONE_MINUS_SRC_ALPHA

Multiplies the operand by the inverted alpha component of the source color.

GL_DEST_ALPHA

Multiplies the operand by the alpha component of the destination color.

GL_ONE_MINUS_DEST_ALPHA

Multiplies the operand by the inverted alpha component of the destination color.

Additionally, the sfactor parameter supports the following:

GL_DST_COLOR

Component-wise multiplication of the operand with the destination color.

GL_ONE_MINUS_DST_COLOR

Component-wise multiplication of the operand with the inverted destination color.

GL_SRC_ALPHA_SATURATE

Returns the minimum of source alpha and inverted destination alpha. This exists mostly for historical reasons, because it was required for an outmoded anti-aliasing technique.

And the dfactor parameter also supports the following:

GL_SRC_COLOR

Component-wise multiplication of the operand with the source color.

GL_ONE_MINUS_SRC_COLOR

Component-wise multiplication of the operand with the inverted source color.

OpenGL ES 2.0 relaxes the blending constraints by unifying the set of choices for sfactor and dfactor, with the exception of GL_SRC_ALPHA_SATURATE.

ES 2.0 also adds the concept of “constant color,” specified via glBlendColor. For more information, look up glBlendColor and glBlendFunc at the Khronos website:

 http://www.khronos.org/opengles/sdk/docs/man/

## Wrangle Premultiplied Alpha

One of the biggest gotchas with textures on Apple devices is the issue of premultiplied alpha. If the RGB components in an image have already been scaled by their associated alpha value, the image is considered to be premultiplied. Normally, PNG images do not store premultiplied RGB values, but Xcode does some tampering with them when it creates the application bundle.

You might recall that we passed in a flag to the CGBitmapInfo mask that’s related to this; Example 6-1 shows a snippet of the ResourceManager class presented in the previous chapter, with the flag of interest highlighted in bold.

Example 6-1. Using a CGContext
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo =
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big;
CGContextRef context = CGBitmapContextCreate(data,
description.Size.x,
description.Size.y,
description.BitsPerComponent,
bpp * description.Size.x,
colorSpace,
bitmapInfo);
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, description.Size.x, description.Size.y);
CGContextDrawImage(context, rect, uiImage.CGImage);
CGContextRelease(context);


For nonpremultiplied alpha, there’s a flag called kCGImageAlphaLast that you’re welcome to try, but at the time of this writing, the Quartz implementation on the iPhone does not support it, and I doubt it ever will, because of the funky preprocessing that Xcode performs on image files.

So, you’re stuck with premultiplied alpha. Don’t panic! There are two rather elegant ways to deal with it:

• Use PVRTexTool to encode your data into a PVR file. Remember, PVRTexTool can encode your image into any OpenGL format; it’s not restricted to the compressed formats.

• Or, adjust your blending equation so that it takes premultiplied alpha into account, like so:

glBlendFunction(GL_ONE, GL_ONE_MINUS_SRC_ALPHA)

By using GL_ONE for the sfactor argument, you’re telling OpenGL there’s no need to multiply the RGB components by alpha.

In the previous chapter, we also presented a method of loading PNG files using CGDataProviderCopyData, but with that technique, the simulator and the device can differ in how they treat alpha. Again, I recommend using PVR files for fast and reliable results.

In Figure 6-1, the left column contains a normal texture, and the right column contains a texture with premultiplied alpha. In every row, the dfactor argument is GL_ONE_MINUS_SRC_ALPHA.

The following list summarizes the best results from Figure 6-1:

• For textures with straight alpha, set sfactor to GL_SRC_ALPHA and dfactor to GL_ONE_MINUS_SRC_ALPHA.

• For textures with premultiplied alpha, set sfactor to GL_ONE and dfactor to GL_ONE_MINUS_SRC_ALPHA.

• To check whether a texture has premultiplied alpha, disable blending, and look at the silhouette.

## Blending Caveats

It’s important to remember to disable depth testing when blending is enabled. If depth testing is turned on, triangles that lie beneath other triangles get completely rejected, so their color can’t contribute to the framebuffer.

An equally important caveat is that you should render your triangles in back-to-front order; the standard blending math simply doesn’t work if you try to draw the top layer before the layer beneath it. Let’s demonstrate why this is so. Suppose you’d like to depict a half-opaque red triangle on top of a half-opaque green triangle. Assuming the clear color is black, the history of a pixel in the framebuffer would look like this if you use back-to-front ordering:

1. Clear to Black. Result: (0, 0, 0).

2. Draw the half-opaque green triangle. Result: (0, 0.5, 0).

3. Draw the half-opaque red triangle. Result: (0.5, 0.25, 0).

So, the resulting pixel is a yellowish red; this is what you’d expect. If you try to draw the red triangle first, the result is different:

1. Clear to Black. Result: (0, 0, 0).

2. Draw the half-opaque red triangle. Result: (0.5, 0, 0).

3. Draw the half-opaque green triangle. Result: (0.25, 0.5, 0).

Now you have yellowish green. Order matters when you’re blending! Incidentally, there’s a way to adjust the blending equations so that you can draw in front-to-back order instead of back-to-front; we’ll show how in the next section.

When blending is enabled, sort your draw calls from farthest to nearest, and disable depth testing.

## Blending Extensions and Their Uses

Always remember to check for extension support using the method described in “Dealing with Size Constraints”. At the time of this writing, the iPhone supports the following blending-related extensions in OpenGL ES 1.1:

GL_OES_blend_subtract (all iPhone models)

Allows you to specify a blending operation other than addition, namely, subtraction.

GL_OES_blend_equation_separate (iPhone 3GS and higher)

Allows you to specify two separate blending operations: one for RGB, the other for alpha.

GL_OES_blend_func_separate (iPhone 3GS and higher)

Allows you to specify two separate pairs of blend factors: one pair for RGB, the other for alpha.

With OpenGL ES 2.0, these extensions are part of the core specification. Together they declare the following functions:

void glBlendEquation(GLenum operation)
void glBlendFuncSeparate(GLenum sfactorRGB, GLenum dfactorRGB,
GLenum sfactorAlpha, GLenum dfactorAlpha);
void glBlendEquationSeparate(GLenum operationRGB, GLenum operationAlpha);

For ES 1.1, remember to append OES to the end of each function since that’s the naming convention for extensions.

The parameters to glBlendEquation and glBlendEquationSeparate can be one of the following:

Adds the source operand to the source operand; this is the default.

GL_FUNC_SUBTRACT

Subtracts the destination operand from the source operand.

GL_FUNC_REVERSE_SUBTRACT

Subtracts the source operand from the destination operand.

Again, remember to append _OES for these constants when working with ES 1.1.

When all these extensions are supported, you effectively have the ability to specify two unique equations: one for alpha, the other for RGB. Each equation conforms to one of the following templates:

FinalColor = SrcColor * sfactor + DestColor * dfactor
FinalColor = SrcColor * sfactor - DestColor * dfactor
FinalColor = DestColor * dfactor - SrcColor * sfactor

### Why Is Blending Configuration Useful?

You might wonder why you’d ever need all the flexibility given by the aforementioned blending extensions. You’ll see how various blending configurations come in handy with some samples presented later in the chapter, but I’ll briefly go over some common uses here.

One use of GL_FUNC_SUBTRACT is inverting a region of color on the screen to highlight it. Simply draw a solid white rectangle and use GL_ONE for both sfactor and dfactor. You could also use subtraction to perform a comparison, or visual “diff,” between two images.

The separate blending equations can be useful too. For example, perhaps you’d like to leave the destination’s alpha channel unperturbed because you’re storing information there for something other than transparency. In such a case, you could say the following:

glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ZERO, GL_ONE);

Another time to use separate blending equations is when you need to draw your triangles in front-to-back order rather than the usual back-to-front order. As you’ll see later in the chapter, this can be useful for certain effects. To pull this off, take the following steps:

1. Set your clear color to (0, 0, 0, 1).

2. Make sure your source texture (or per-vertex color) has premultiplied alpha.

3. Set your blend equation to the following:

glBlendFuncSeparate(GL_DST_ALPHA, GL_ONE, GL_ZERO, GL_ONE_MINUS_SRC_ALPHA);

To see why this works, let’s go back to the example of a half-opaque red triangle being rendered on top of a half-opaque green triangle:

1. Clear to Black. Result: (0, 0, 0, 1).

2. Draw the half-opaque red triangle. Since it’s premultiplied, its source color is (0.5, 0, 0, 0.5). Using the previous blending equation, the result is (0.5, 0, 0, 0.5).

3. Draw the half-opaque green triangle; its source color is (0, 0.5, 0, 0.5). The result after blending is (0.5, 0.25, 0, 0.25).

The resulting pixel is yellowish red, just as you’d expect. Note that the framebuffer’s alpha value is always inverted when you’re using this trick.

## Shifting Texture Color with Per-Vertex Color

Sometimes you’ll need to uniformly tweak the alpha values across an entire texture. For example, you may want to create a fade-in effect or make a texture semitransparent for drawing a heads-up display (HUD).

With OpenGL ES 1.1, this can be achieved simply by adjusting the current vertex color:

glColor4f(1, 1, 1, alpha);

By default, OpenGL multiplies each component of the current vertex color with the color of the texel that it’s rendering. This is known as modulation, and it’s actually only one of many ways that you can combine texture color with per-vertex color (this will be discussed in detail later in this book).

If you’re using a texture with premultiplied alpha, then the vertex color should also be premultiplied. The aforementioned function call should be changed to the following:

glColor4f(alpha, alpha, alpha, alpha);

Sometimes you may want to throttle back only one color channel. For example, say your app needs to render some red and blue buttons and that all the buttons are identical except for their color. Rather than wasting memory with multiple texture objects, you can create a single grayscale texture and modulate its color, like this:

// Bind the grayscale button texture.
glBindTexture(GL_TEXTURE_2D, buttonTexture)

// Draw green button.
glColor4f(0, 1, 0, 1);
glDrawElements(...);

// Draw red button.
glColor4f(1, 0, 0, 1);
glDrawElements(...);

With ES 2.0, the modulation needs to be performed within the pixel shader itself:

varying lowp vec4 Color;
varying mediump vec2 TextureCoord;

uniform sampler2D Sampler;

void main(void)
{
gl_FragColor = texture2D(Sampler, TextureCoord) * Color;
}


The previous code snippet should look familiar. We used the same technique in Chapter 5 when combining lighting color with texture color.

## Poor Man’s Reflection with the Stencil Buffer

One use for blending in a 3D scene is overlaying a reflection on top of a surface, as shown on the left of Figure 6-2. Remember, computer graphics is often about cheating! To create the reflection, you can redraw the object using an upside-down projection matrix. Note that you need a way to prevent the reflection from “leaking” outside the bounds of the reflective surface, as shown on the right in Figure 6-2. How can this be done?

It turns out that third-generation iPhones and iPod touches have support for an OpenGL ES feature known as the stencil buffer, and it’s well-suited to this problem. The stencil buffer is actually just another type of renderbuffer, much like color and depth. But instead of containing RGB or Z values, it holds a small integer value at every pixel that you can use in different ways. There are many applications for the stencil buffer beyond clipping.

To accommodate older iPhones, we’ll cover some alternatives to stenciling later in the chapter.

To check whether stenciling is supported on the iPhone, check for the GL_OES_stencil8 extension using the method in “Dealing with Size Constraints”. At the time of this writing, stenciling is supported on third-generation devices and the simulator, but not on first- and second-generation devices.

The reflection trick can be achieved in four steps (see Figure 6-3):

1. Render the disk to stencil only.

2. Render the reflection of the floating object with the stencil test enabled.

3. Clear the depth buffer, and render the actual floating object.

4. Render the disk using front-to-back blending.

Note that the reflection is drawn before the textured podium, which is the reason for the front-to-back blending. We can’t render the reflection after the podium because blending and depth-testing cannot both be enabled when drawing complex geometry.

The complete code for this sample is available from this book’s website, but we’ll go over the key snippets in the following subsections. First let’s take a look at the creation of the stencil buffer itself. The first few steps are generating a renderbuffer identifier, binding it, and allocating storage. This may look familiar if you remember how to create the depth buffer:

GLuint stencil;
glGenRenderbuffersOES(1, &stencil);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, stencil);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_STENCIL_INDEX8_OES, width, height);


Next, attach the stencil buffer to the framebuffer object, shown in bold here:

GLuint framebuffer;
glGenFramebuffersOES(1, &framebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, framebuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES, color);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES,
GL_RENDERBUFFER_OES, depth);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_STENCIL_ATTACHMENT_OES,
GL_RENDERBUFFER_OES, stencil);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, color);


As always, remember to omit the OES endings when working with ES 2.0.

To save memory, sometimes you can interleave the depth buffer and stencil buffer into a single renderbuffer. This is possible only when the OES_packed_depth_stencil extension is supported. At the time of this writing, it’s available on third-generation devices, but not on the simulator or older devices. To see how to use this extension, see Example 6-2. Relevant portions are highlighted in bold.

Example 6-2. Using packed depth stencil
GLuint depthStencil;
glGenRenderbuffersOES(1, &depthStencil);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, depthStencil);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH24_STENCIL8_OES, width, height);

GLuint framebuffer;
glGenFramebuffersOES(1, &framebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, framebuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES, color);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES,
GL_RENDERBUFFER_OES, depthStencil);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_STENCIL_ATTACHMENT_OES,
GL_RENDERBUFFER_OES, depthStencil);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, color);


### Rendering the Disk to Stencil Only

Recall that step 1 in our reflection demo renders the disk to the stencil buffer. Before drawing to the stencil buffer, it needs to be cleared, just like any other renderbuffer:

glClearColor(0, 0, 0, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

Next you need to tell OpenGL to enable writes to the stencil buffer, and you need to tell it what stencil value you’d like to write. Since you’re using an 8-bit buffer in this case, you can set any value between 0x00 and 0xff. Let’s go with 0xff and set up the OpenGL state like this:

glEnable(GL_STENCIL_TEST);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 0xff, 0xff);


The first line enables GL_STENCIL_TEST, which is a somewhat misleading name in this case; you’re writing to the stencil buffer, not testing against it. If you don’t enable GL_STENCIL_TEST, then OpenGL assumes you’re not working with the stencil buffer at all.

The next line, glStencilOp, tells OpenGL which stencil operation you’d like to perform at each pixel. Here’s the formal declaration:

void glStencilOp(GLenum fail, GLenum zfail, GLenum zpass);

GLenum fail

Specifies the operation to perform when the stencil test fails

GLenum zfail

Specifies the operation to perform when the stencil test passes and the depth test fails

GLenum zpass

Specifies the operation to perform when the stencil test passes and the depth test passes

Since the disk is the first draw call in the scene, we don’t care whether any of these tests fail, so we’ve set them all to the same value.

Each of the arguments to glStencilOp can be one of the following:

GL_REPLACE

Replace the value that’s currently in the stencil buffer with the value specified in glStencilFunc.

GL_KEEP

Don’t do anything.

GL_INCR

Increment the value that’s currently in the stencil buffer.

GL_DECR

Decrement the value that’s currently in the stencil buffer.

GL_INVERT

Perform a bitwise NOT operation with the value that’s currently in the stencil buffer.

GL_ZERO

Clobber the current stencil buffer value with zero.

Again, this may seem like way too much flexibility, more than you’d ever need. Later in this book, you’ll see how all this freedom can be used to perform interesting tricks. For now, all we’re doing is writing the shape of the disk out to the stencil buffer, so we’re using the GL_REPLACE operation.

The next function we called to set up our stencil state is glStencilFunc. Here’s its function declaration:

void glStencilFunc(GLenum func, GLint ref, GLuint mask);

GLenum func

This specifies the comparison function to use for the stencil test, much like the depth test “Creating and Using the Depth Buffer”.

GLint ref

This “reference value” actually serves two purposes:

• Comparison value to test against if func is something other than GL_ALWAYS or GL_NEVER

• The value to write if the operation is GL_REPLACE

Before performing a comparison, this bitmask gets ANDed with both the reference value and the value that’s already in the buffer.

Again, this gives the developer quite a bit of power, but in this case we only need something simple.

Getting back to the task at hand, check out Example 6-3 to see how to render the disk to the stencil buffer only. I adjusted the indentation of the code to show how certain pieces of OpenGL state get modified before the draw call and then restored after the draw call.

Example 6-3. Rendering the disk to stencil only
// Prepare the render state for the disk.
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 0xff, 0xff);

// Render the disk to the stencil buffer only.
glDisable(GL_TEXTURE_2D);
glTranslatef(0, DiskY, 0);
RenderDrawable(m_drawables.Disk); // private method that calls glDrawElements
glTranslatef(0, -DiskY, 0);
glEnable(GL_TEXTURE_2D);


Two new function calls appear in Example 6-3: glDepthMask and glColorMask. Recall that we’re interested in affecting values in the stencil buffer only. It’s actually perfectly fine to write to all three renderbuffers (color, depth, stencil), but to maximize performance, it’s good practice to disable any writes that you don’t need.

The four arguments to glColorMask allow you to toggle each of the individual color channels; in this case we don’t need any of them. Note that glDepthMask has only one argument, since it’s a single-component buffer. Incidentally, OpenGL ES also provides a glStencilMask function, which we’re not using here.

### Rendering the Reflected Object with Stencil Testing

Step 2 renders the reflection of the object and uses the stencil buffer to clip it to the boundary of the disk. Example 6-4 shows how to do this.

Example 6-4. Rendering the reflection
glTranslatef(0, KnotY, 0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_EQUAL, 0xff, 0xff);
glEnable(GL_LIGHTING);
glBindTexture(GL_TEXTURE_2D, m_textures.Grille);

const float alpha = 0.4f;
vec4 diffuse(alpha, alpha, alpha, 1 - alpha);
glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, diffuse.Pointer());

glMatrixMode(GL_PROJECTION);
RenderDrawable(m_drawables.Knot); // private method that calls glDrawElements
glMatrixMode(GL_MODELVIEW);


This time we don’t need to change the values in the stencil buffer, so we use GL_KEEP for the argument to glStencilOp. We changed the stencil comparison function to GL_EQUAL so that only the pixels within the correct region will pass.

There are several ways you could go about drawing an object upside down, but I chose to do it with a quick-and-dirty projection matrix. The result isn’t a very accurate reflection, but it’s good enough to fool the viewer! Example 6-5 shows how I did this using a mat4 method from the C++ vector library in the appendix. (For ES 1.1, you could simply use the provided glFrustum function.)

Example 6-5. Computing two projection matrices
const float AspectRatio = (float) height / width;
const float Shift = -1.25;
const float Near = 5;
const float Far = 50;

m_projection = mat4::Frustum(-1, 1,
-AspectRatio, AspectRatio,
Near, Far);

m_mirrorProjection = mat4::Frustum(-1, 1,
AspectRatio + Shift, -AspectRatio + Shift,
Near, Far);


### Rendering the “Real” Object

The next step is rather mundane; we simply need to render the actual floating object, without doing anything with the stencil buffer. Before calling glDrawElements for the object, we turn off the stencil test and disable the depth buffer:

glDisable(GL_STENCIL_TEST);
glClear(GL_DEPTH_BUFFER_BIT);


For the first time, we’ve found a reason to call glClear somewhere in the middle of the Render method! Importantly, we’re clearing only the depth buffer, leaving the color buffer intact.

Remember, the reflection is drawn just like any other 3D object, complete with depth testing. Allowing the actual object to be occluded by the reflection would destroy the illusion, so it’s a good idea to clear the depth buffer before drawing it. Given the fixed position of the camera in our demo, we could actually get away without performing the clear, but this allows us to tweak the demo without breaking anything.

### Rendering the Disk with Front-to-Back Blending

The final step is rendering the marble disk underneath the reflection. Example 6-6 sets this up.

Example 6-6. Render the disk to the color buffer
glTranslatef(0, DiskY - KnotY, 0);
glDisable(GL_LIGHTING);
glBindTexture(GL_TEXTURE_2D, m_textures.Marble);
glBlendFuncSeparateOES(GL_DST_ALPHA, GL_ONE,             // RGB factors
GL_ZERO, GL_ONE_MINUS_SRC_ALPHA); // Alpha factors
glEnable(GL_BLEND);


## Stencil Alternatives for Older iPhones

If your app needs to accommodate first- and second-generation iPhones, in many cases you can use a trick that acts like stenciling without actually requiring a stencil buffer. These various tricks include the following:

• Using the framebuffer’s alpha component to store the “stencil” values and setting up a blending equation that tests against those values.

• Turning off color writes and writing to the depth buffer to mask out certain regions. (The easiest way to uniformly offset generated depth values is with the glDepthRange function.)

• Cropping simple rectangular regions can be achieved with OpenGL’s glScissor function.

• Some of the bitwise operations available with stencil buffers are actually possible with colors as well. In fact, there are additional operations possible with colors, such as XOR. To see how to do this, check out the glLogicOp function.

Let’s demonstrate the first trick in the previous list: using framebuffer alpha as a fake stencil buffer. With this technique, it’s possible to achieve the result shown in Figure 6-2 on older iPhones. The sequence of operations becomes the following:

1. Clear the depth buffer.

2. Render the background image with α = 0.

3. Render the textured disk normally with α = 1.

4. Enable blending, and set the blending equation to S*Dα+D*(1 – Dα).

5. Render the reflection of the floating object.

6. Set the blending equation to S*Sα+D*(1 – Sα).

7. Turn off depth testing, and render the textured disk again with α = 0.5; this fades out the reflection a bit.

8. Clear the depth buffer, and re-enable depth testing.

9. Render the actual floating object.

Example 6-7 shows the rendering code for these nine steps. As always, the entire sample code is available from this book’s website.

Example 6-7. Faking the stencil buffer
glClear(GL_DEPTH_BUFFER_BIT);

// Set up the transforms for the background.
glMatrixMode(GL_PROJECTION);
glFrustumf(-0.5, 0.5, -0.5, 0.5, NearPlane, FarPlane);
glMatrixMode(GL_MODELVIEW);
glTranslatef(0, 0, -NearPlane * 2);

// Render the dark background with alpha = 0.
glDisable(GL_DEPTH_TEST);
glColor4f(0.5, 0.5, 0.5, 0);
glBindTexture(GL_TEXTURE_2D, m_textures.Tiger);

// Set up the transforms for the 3D scene.
glMatrixMode(GL_PROJECTION);
glMatrixMode(GL_MODELVIEW);
glRotatef(20, 1, 0, 0);
glBindTexture(GL_TEXTURE_2D, m_textures.Marble);

// Render the disk normally.
glColor4f(1, 1, 1, 1);
glTranslatef(0, DiskY, 0);
RenderDrawable(m_drawables.Disk);
glTranslatef(0, -DiskY, 0);
glEnable(GL_DEPTH_TEST);

// Render the reflection.
glPushMatrix();
glRotatef(theta, 0, 1, 0);
glTranslatef(0, KnotY, 0);
glEnable(GL_LIGHTING);
glBindTexture(GL_TEXTURE_2D, m_textures.Grille);
glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA);
glEnable(GL_BLEND);
glMatrixMode(GL_PROJECTION);
RenderDrawable(m_drawables.Knot);
glMatrixMode(GL_MODELVIEW);
glDisable(GL_LIGHTING);
glPopMatrix();

// Render the disk again to make the reflection fade out.
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glBindTexture(GL_TEXTURE_2D, m_textures.Marble);
glColor4f(1, 1, 1, 0.5);
glDisable(GL_DEPTH_TEST);
glTranslatef(0, DiskY, 0);
RenderDrawable(m_drawables.Disk);
glTranslatef(0, -DiskY, 0);
glEnable(GL_DEPTH_TEST);
glColor4f(1, 1, 1, 1);
glDisable(GL_BLEND);

// Clear the depth buffer.
glClear(GL_DEPTH_BUFFER_BIT);

// Render the floating object.
glEnable(GL_LIGHTING);
glBindTexture(GL_TEXTURE_2D, m_textures.Grille);
glPushMatrix();
glTranslatef(0, KnotY, 0);
glRotatef(theta, 0, 1, 0);
RenderDrawable(m_drawables.Knot);
glPopMatrix();
glDisable(GL_LIGHTING);

## Anti-Aliasing Tricks with Offscreen FBOs

The iPhone’s first-class support for framebuffer objects is perhaps its greatest enabler of unique effects. In every sample presented so far in this book, we’ve been using a single FBO, namely, the FBO that represents the visible Core Graphics layer. It’s important to realize that FBOs can also be created as offscreen surfaces, meaning they don’t show up on the screen unless bound to a texture. In fact, on most platforms, FBOs are always offscreen. The iPhone is rather unique in that the visible layer is itself treated as an FBO (albeit a special one).

Binding offscreen FBOs to textures enables a whole slew of interesting effects, including page-curling animations, light blooming, and more. We’ll cover some of these techniques later in this book, but recall that one of the topics of this chapter is anti-aliasing. Several sneaky tricks with FBOs can be used to achieve full-scene anti-aliasing, even though the iPhone does not directly support anti-aliasing! We’ll cover two of these techniques in the following subsections.

One technique not discussed here is performing a postprocess on the final image to soften it. While this is not true anti-aliasing, it may produce good results in some cases. It’s similar to the bloom effect covered in Chapter 8.

### A Super Simple Sample App for Supersampling

The easiest and crudest way to achieve full-scene anti-aliasing on the iPhone is to leverage bilinear texture filtering. Simply render to an offscreen FBO that has twice the dimensions of the screen, and then bind it to a texture and scale it down, as shown in Figure 6-4. This technique is known as supersampling.

To demonstrate how to achieve this effect, we’ll walk through the process of extending the stencil sample to use supersampling. As an added bonus, we’ll throw in an Apple-esque flipping animation, as shown in Figure 6-5. Since we’re creating a secondary FBO anyway, flipping effects like this come virtually for free.

Example 6-8 shows the RenderingEngine class declaration and related type definitions. Class members that carry over from previous samples are replaced with an ellipses for brevity.

Example 6-8. RenderingEngine declaration for the anti-aliasing sample
struct Framebuffers {
GLuint Small;
GLuint Big;
};

struct Renderbuffers {
GLuint SmallColor;
GLuint BigColor;
GLuint BigDepth;
GLuint BigStencil;
};

struct Textures {
GLuint Marble;
GLuint RhinoBackground;
GLuint TigerBackground;
GLuint OffscreenSurface;
};

class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize();
void Render(float objectTheta, float fboTheta) const;
private:
ivec2 GetFboSize() const;
Textures m_textures;
Renderbuffers m_renderbuffers;
Framebuffers m_framebuffers;
// ...
};

 The “small” FBO is attached to the visible EAGL layer (320×480). The “big” FBO is the 640×960 surface that contains the 3D scene. The small FBO does not need depth or stencil attachments because the only thing it contains is a full-screen quad; the big FBO is where most of the 3D rendering takes place, so it needs depth and stencil. The 3D scene requires a marble texture for the podium and one background for each side of the animation (Figure 6-5). The fourth texture object, OffscreenSurface, is attached to the big FBO. The application layer passes in objectTheta to control the rotation of the podium and passes in fboTheta to control the flipping transitions. GetFboSize is a new private method for conveniently determining the size of the currently bound FBO. This method helps avoid the temptation to hardcode some magic numbers or to duplicate state that OpenGL already maintains.

First let’s take a look at the GetFboSize implementation (Example 6-9), which returns a width-height pair for the size. The return type is an instance of ivec2, one of the types defined in the C++ vector library in the appendix.

Example 6-9. GetFboSize() implementation
ivec2 RenderingEngine::GetFboSize() const
{
ivec2 size;
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_WIDTH_OES, &size.x);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_HEIGHT_OES, &size.y);
return size;
}

Next let’s deal with the creation of the two FBOs. Recall the steps for creating the on-screen FBO used in almost every sample so far:

1. In the RenderingEngine constructor, generate an identifier for the color renderbuffer, and then bind it to the pipeline.

2. In the GLView class (Objective-C), allocate storage for the color renderbuffer like so:

[m_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:eaglLayer]
3. In the RenderingEngine::Initialize method, create a framebuffer object, and attach the color renderbuffer to it.

4. If desired, create and allocate renderbuffers for depth and stencil, and then attach them to the FBO.

For the supersampling sample that we’re writing, we still need to perform the first three steps in the previous sequence, but then we follow it with the creation of the offscreen FBO. Unlike the on-screen FBO, its color buffer is allocated in much the same manner as depth and stencil:

glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_RGBA8_OES, width, height);

See Example 6-10 for the Initialize method used in the supersampling sample.

Example 6-10. Initialize() for supersampling
void RenderingEngine::Initialize()
{
// Create the on-screen FBO.

glGenFramebuffersOES(1, &m_framebuffers.Small);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Small);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.SmallColor);

// Create the double-size off-screen FBO.

ivec2 size = GetFboSize() * 2;

glGenRenderbuffersOES(1, &m_renderbuffers.BigColor);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.BigColor);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_RGBA8_OES,
size.x, size.y);

glGenRenderbuffersOES(1, &m_renderbuffers.BigDepth);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.BigDepth);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH_COMPONENT24_OES,
size.x, size.y);

glGenRenderbuffersOES(1, &m_renderbuffers.BigStencil);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.BigStencil);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_STENCIL_INDEX8_OES,
size.x, size.y);

glGenFramebuffersOES(1, &m_framebuffers.Big);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Big);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.BigColor);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_DEPTH_ATTACHMENT_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.BigDepth);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_STENCIL_ATTACHMENT_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.BigStencil);

// Create a texture object and associate it with the big FBO.

glGenTextures(1, &m_textures.OffscreenSurface);
glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenSurface);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x, size.y, 0,
GL_RGBA, GL_UNSIGNED_BYTE, 0);
glFramebufferTexture2DOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES,
GL_TEXTURE_2D, m_textures.OffscreenSurface, 0);

// Check FBO status.

GLenum status = glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES);
if (status != GL_FRAMEBUFFER_COMPLETE_OES) {
cout << "Incomplete FBO" << endl;
exit(1);
}

// Load textures, create VBOs, set up various GL state.
...
}


You may have noticed two new FBO-related function calls in Example 6-10: glFramebufferTexture2DOES and glCheckFramebufferStatusOES. The formal function declarations look like this:

void glFramebufferTexture2DOES(GLenum target,
GLenum attachment, GLenum textarget,
GLuint texture, GLint level);

GLenum glCheckFramebufferStatusOES(GLenum target);


(As usual, the OES suffix can be removed for ES 2.0.)

The glFramebufferTexture2DOES function allows you to cast a color buffer into a texture object. FBO texture objects get set up just like any other texture object: they have an identifier created with glGenTextures, they have filter and wrap modes, and they have a format that should match the format of the FBO. The main difference with FBO textures is the fact that null gets passed to the last argument of glTexImage2D, since there’s no image data to upload.

Note that the texture in Example 6-10 has non-power-of-two dimensions, so it specifies clamp-to-edge wrapping to accommodate third-generation devices. For older iPhones, the sample won’t work; you’d have to change it to POT dimensions. Refer to “Dealing with Size Constraints” for hints on how to do this. Keep in mind that the values passed to glViewport need not match the size of the renderbuffer; this comes in handy when rendering to an NPOT subregion of a POT texture.

The other new function, glCheckFramebufferStatusOES, is a useful sanity check to make sure that an FBO has been set up properly. It’s easy to bungle the creation of FBOs if the sizes of the attachments don’t match up or if their formats are incompatible with each other. glCheckFramebufferStatusOES returns one of the following values, which are fairly self-explanatory:

• GL_FRAMEBUFFER_COMPLETE

• GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT

• GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT

• GL_FRAMEBUFFER_INCOMPLETE_DIMENSIONS

• GL_FRAMEBUFFER_INCOMPLETE_FORMATS

• GL_FRAMEBUFFER_UNSUPPORTED

Next let’s take a look at the render method of the supersampling sample. Recall from the class declaration that the application layer passes in objectTheta to control the rotation of the podium and passes in fboTheta to control the flipping transitions. So, the first thing the Render method does is look at fboTheta to determine which background image should be displayed and which shape should be shown on the podium. See Example 6-11.

Example 6-11. Render() for supersampling
void RenderingEngine::Render(float objectTheta, float fboTheta) const
{
Drawable drawable;
GLuint background;
vec3 color;

// Look at fboTheta to determine which "side" should be rendered:
//   1) Orange Trefoil knot against a Tiger background
//   2) Green Klein bottle against a Rhino background

if (fboTheta > 270 || fboTheta < 90) {
background = m_textures.TigerBackground;
drawable = m_drawables.Knot;
color = vec3(1, 0.5, 0.1);
} else {
background = m_textures.RhinoBackground;
drawable = m_drawables.Bottle;
color = vec3(0.5, 0.75, 0.1);
}

// Bind the double-size FBO.
glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Big);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.BigColor);
ivec2 bigSize = GetFboSize();
glViewport(0, 0, bigSize.x, bigSize.y);

// Draw the 3D scene - download the example to see this code.
...

// Render the background.
glColor4f(0.7, 0.7, 0.7, 1);
glBindTexture(GL_TEXTURE_2D, background);
glMatrixMode(GL_PROJECTION);
glFrustumf(-0.5, 0.5, -0.5, 0.5, NearPlane, FarPlane);
glMatrixMode(GL_MODELVIEW);
glTranslatef(0, 0, -NearPlane * 2);
glColor4f(1, 1, 1, 1);
glDisable(GL_BLEND);

// Switch to the on-screen render target.
glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Small);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.SmallColor);
ivec2 smallSize = GetFboSize();
glViewport(0, 0, smallSize.x, smallSize.y);

// Clear the color buffer only if necessary.
if ((int) fboTheta % 180 != 0) {
glClearColor(0, 0, 0, 1);
glClear(GL_COLOR_BUFFER_BIT);
}

// Render the offscreen surface by applying it to a quad.
glDisable(GL_DEPTH_TEST);
glRotatef(fboTheta, 0, 1, 0);
glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenSurface);
glDisable(GL_TEXTURE_2D);
}


Most of Example 6-11 is fairly straightforward. One piece that may have caught your eye is the small optimization made right before blitting the offscreen FBO to the screen:

// Clear the color buffer only if necessary.
if ((int) fboTheta % 180 != 0) {
glClearColor(0, 0, 0, 1);
glClear(GL_COLOR_BUFFER_BIT);
}

This is a sneaky little trick. Since the quad is the exact same size as the screen, there’s no need to clear the color buffer; unnecessarily issuing a glClear can hurt performance. However, if a flipping animation is currently underway, the color buffer needs to be cleared to prevent artifacts from appearing in the background; flip back to Figure 6-5 and observe the black areas. If fboTheta is a multiple of 180, then the quad completely fills the screen, so there’s no need to issue a clear.

That’s it for the supersampling sample. The quality of the anti-aliasing is actually not that great; you can still see some “stair-stepping” along the bottom outline of the shape in Figure 6-6. You might think that creating an even bigger offscreen buffer, say quadruple-size, would provide higher-quality results. Unfortunately, using a quadruple-size buffer would require two passes; directly applying a 1280×1920 texture to a 320×480 quad isn’t sufficient because GL_LINEAR filtering only samples from a 2×2 neighborhood of pixels. To achieve the desired result, you’d actually need three FBOs as follows:

• 1280×1920 offscreen FBO for the 3D scene

• 640×960 offscreen FBO that contains a quad with the 1280×1920 texture applied to it

• 320×480 on-screen FBO that contains a quad with the 640×960 texture applied to it

Not only is this laborious, but it’s a memory hog. Older iPhones don’t even support textures this large! It turns out there’s another anti-aliasing strategy called jittering, and it can produce high-quality results without the memory overhead of supersampling.

### Jittering

Jittering is somewhat more complex to implement than supersampling, but it’s not rocket science. The idea is to rerender the scene multiple times at slightly different viewpoints, merging the results along the way. You need only two FBOs for this method: the on-screen FBO that accumulates the color and the offscreen FBO that the 3D scene is rendered to. You can create as many jittered samples as you’d like, and you still need only two FBOs. Of course, the more jittered samples you create, the longer it takes to create the final rendering. Example 6-12 shows the pseudocode for the jittering algorithm.

Example 6-12. Jitter pseudocode
BindFbo(OnscreenBuffer)
glClear(GL_COLOR_BUFFER_BIT)

for (int sample = 0; sample < SampleCount; sample++) {
BindFbo(OffscreenBuffer)

vec2 offset = JitterTable[sample]

SetFrustum(LeftPlane + offset.x, RightPlane + offset.x,
TopPlane + offset.y, BottomPlane + offset.y,
NearPlane, FarPlane)

Render3DScene()

f = 1.0 / SampleCount
glColor4f(f, f, f, 1)
glEnable(GL_BLEND)
glBlendFunc(GL_ONE, GL_ONE)

BindFbo(OnscreenBuffer)
BindTexture(OffscreenBuffer)
}


The key part of Example 6-12 is the blending configuration. By using a blend equation of plain old addition (GL_ONE, GL_ONE) and dimming the color according to the number of samples, you’re effectively accumulating an average color.

An unfortunate side effect of jittering is reduced color precision; this can cause banding artifacts, as shown in Figure 6-7. On some platforms the banding effect can be neutralized with a high-precision color buffer, but that’s not supported on the iPhone. In practice, I find that creating too many samples is detrimental to performance anyway, so the banding effect isn’t usually much of a concern.

Determining the jitter offsets (JitterTable in Example 6-12) is a bit of black art. Totally random values don’t work well since they don’t guarantee uniform spacing between samples. Interestingly, dividing up each pixel into an equally spaced uniform grid does not work well either! Example 6-13 shows some commonly used jitter offsets.

Example 6-13. Popular jitter offsets
const vec2 JitterOffsets2[2] =
{
vec2(0.25f, 0.75f), vec2(0.75f, 0.25f),
};

const vec2 JitterOffsets4[4] =
{
vec2(0.375f, 0.25f), vec2(0.125f, 0.75f),
vec2(0.875f, 0.25f), vec2(0.625f, 0.75f),
};

const vec2 JitterOffsets8[8] =
{
vec2(0.5625f, 0.4375f), vec2(0.0625f, 0.9375f),
vec2(0.3125f, 0.6875f), vec2(0.6875f, 0.8125f),

vec2(0.8125f, 0.1875f), vec2(0.9375f, 0.5625f),
vec2(0.4375f, 0.0625f), vec2(0.1875f, 0.3125f),
};

const vec2 JitterOffsets16[16] =
{
vec2(0.375f, 0.4375f), vec2(0.625f, 0.0625f),
vec2(0.875f, 0.1875f), vec2(0.125f, 0.0625f),

vec2(0.375f, 0.6875f), vec2(0.875f, 0.4375f),
vec2(0.625f, 0.5625f), vec2(0.375f, 0.9375f),

vec2(0.625f, 0.3125f), vec2(0.125f, 0.5625f),
vec2(0.125f, 0.8125f), vec2(0.375f, 0.1875f),

vec2(0.875f, 0.9375f), vec2(0.875f, 0.6875f),
vec2(0.125f, 0.3125f), vec2(0.625f, 0.8125f),
};


Let’s walk through the process of creating a simple app with jittering. Much like we did with the supersample example, we’ll include a fun transition animation. (You can download the full project from the book’s website at http://oreilly.com/catalog/9780596804831.) This time we’ll use the jitter offsets to create a defocusing effect, as shown in Figure 6-8.

To start things off, let’s take a look at the RenderingEngine class declaration and related types. It’s not unlike the class we used for supersampling; the main differences are the labels we give to the FBOs. Accumulated denotes the on-screen buffer, and Scene denotes the offscreen buffer. See Example 6-14.

Example 6-14. RenderingEngine declaration for the jittering sample
struct Framebuffers {
GLuint Accumulated;
GLuint Scene;
};

struct Renderbuffers {
GLuint AccumulatedColor;
GLuint SceneColor;
GLuint SceneDepth;
GLuint SceneStencil;
};

struct Textures {
GLuint Marble;
GLuint RhinoBackground;
GLuint TigerBackground;
GLuint OffscreenSurface;
};

class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize();
void Render(float objectTheta, float fboTheta) const;
private:
void RenderPass(float objectTheta, float fboTheta, vec2 offset) const;
Textures m_textures;
Renderbuffers m_renderbuffers;
Framebuffers m_framebuffers;
// ...
};


Example 6-14 also adds a new private method called RenderPass; the implementation is shown in Example 6-15. Note that we’re keeping the fboTheta argument that we used in the supersample example, but now we’re using it to compute a scale factor for the jitter offset rather than a y-axis rotation. If fboTheta is 0 or 180, then the jitter offset is left unscaled, so the scene is in focus.

Example 6-15. RenderPass method for jittering
void RenderingEngine::RenderPass(float objectTheta, float fboTheta, vec2 offset) const
{
// Tweak the jitter offset for the defocus effect:

offset -= vec2(0.5, 0.5);
offset *= 1 + 100 * sin(fboTheta * Pi / 180);

// Set up the frustum planes:

const float AspectRatio = (float) m_viewport.y / m_viewport.x;
const float NearPlane = 5;
const float FarPlane = 50;
const float LeftPlane = -1;
const float RightPlane = 1;
const float TopPlane = -AspectRatio;
const float BottomPlane = AspectRatio;

// Transform the jitter offset from window space to eye space:

offset.x *= (RightPlane - LeftPlane) / m_viewport.x;
offset.y *= (BottomPlane - TopPlane) / m_viewport.y;

// Compute the jittered projection matrix:

mat4 projection = mat4::Frustum(LeftPlane + offset.x,
RightPlane + offset.x,
TopPlane + offset.y,
BottomPlane + offset.y,
NearPlane, FarPlane);

// Render the 3D scene - download the example to see this code.
...
}


Example 6-16 shows the implementation to the main Render method. The call to RenderPass is shown in bold.

Example 6-16. Render method for jittering
void RenderingEngine::Render(float objectTheta, float fboTheta) const
{

// This is where you put the jitter offset declarations
// from Example 6-13.

const int JitterCount = 8;
const vec2* JitterOffsets = JitterOffsets8;

glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Accumulated);
glBindRenderbufferOES(GL_RENDERBUFFER_OES,
m_renderbuffers.AccumulatedColor);

glClearColor(0, 0, 0, 1);
glClear(GL_COLOR_BUFFER_BIT);

for (int i = 0; i < JitterCount; i++) {

glBindFramebufferOES(GL_FRAMEBUFFER_OES, m_framebuffers.Scene);
glBindRenderbufferOES(GL_RENDERBUFFER_OES,
m_renderbuffers.SceneColor);

RenderPass(objectTheta,
fboTheta, JitterOffsets[i]);

glMatrixMode(GL_PROJECTION);

const float NearPlane = 5, FarPlane = 50;
glFrustumf(-0.5, 0.5, -0.5, 0.5, NearPlane, FarPlane);

glMatrixMode(GL_MODELVIEW);
glTranslatef(0, 0, -NearPlane * 2);

float f = 1.0f / JitterCount;
f *= (1 + abs(sin(fboTheta * Pi / 180)));
glColor4f(f, f, f, 1);

glEnable(GL_BLEND);
glBlendFunc(GL_ONE, GL_ONE);
glBindFramebufferOES(GL_FRAMEBUFFER_OES,
m_framebuffers.Accumulated);
glBindRenderbufferOES(GL_RENDERBUFFER_OES,
m_renderbuffers.AccumulatedColor);
glDisable(GL_DEPTH_TEST);
glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenSurface);
glDisable(GL_TEXTURE_2D);
glDisable(GL_BLEND);
}
}


Example 6-16 might give you sense of déjà vu; it’s basically an implementation of the pseudocode algorithm that we already presented in Example 6-12. One deviation is how we compute the dimming effect:

float f = 1.0f / JitterCount;
f *= (1 + abs(sin(fboTheta * Pi / 180)));
glColor4f(f, f, f, 1);


The second line in the previous snippet is there only for the special transition effect. In addition to defocusing the scene, it’s also brightened to simulate pupil dilation. If fboTheta is 0 or 180, then f is left unscaled, so the scene has its normal brightness.

### Other FBO Effects

An interesting variation on jittering is depth of field, which blurs out the near and distant portions of the scene. To pull this off, compute the viewing frustum such that a given slice (parallel to the viewing plane) stays the same with each jitter pass; this is the focus plane.

Yet another effect is motion blur, which simulates the ghosting effect seen on displays with low response times. With each pass, make incremental adjustments to your animation, and gradually fade in the alpha value using glColor.

## Rendering Anti-Aliased Lines with Textures

Sometimes full-screen anti-aliasing is more than you really need and can cause too much of a performance hit. You may find that you need anti-aliasing only on your line primitives rather than the entire scene. Normally this would be achieved in OpenGL ES like so:

glEnable(GL_LINE_SMOOTH);

Alas, none of the iPhone models supports this at the time of this writing. However, the simulator does support line smoothing; watch out for inconsistencies like this!

A clever trick to work around this limitation is filling an alpha texture with a circle and then tessellating the lines into short triangle strips (Figure 6-9). Texture coordinates are chosen such that the circle is stretched in the right places. That has the added benefit of allowing round end-cap styles and wide lines.

Using a 16×16 circle for the texture works well for thick lines (see the left circle in Figure 6-9 and left panel in Figure 6-10). For thinner lines, I find that a highly blurred 16x16 texture produces good results (see the right circle in Figure 6-9 and right panel in Figure 6-10).

Let’s walk through the process of converting a line list into a textured triangle list. Each source vertex needs to be extruded into four new vertices. It helps to give each extrusion vector a name using cardinal directions, as shown in Figure 6-11.

Before going over the extrusion algorithm, let’s set up an example scenario. Say we’re rendering an animated stick figure similar to Figure 6-10. Note that some vertices are shared by multiple lines, so it makes sense to use an index buffer. Suppose the application can render the stick figure using either line primitives or textured triangles. Let’s define a StickFigure structure that stores the vertex and index data for either the non-AA variant or the AA variant; see Example 6-17. The non-AA variant doesn’t need texture coordinates, but we’re including them for simplicity’s sake.

Example 6-17. Structures for the extrusion algorithm
struct Vertex {
vec3 Position;
vec2 TexCoord;
};

typedef std::vector<Vertex> VertexList;
typedef std::vector<GLushort> IndexList;

struct StickFigure {
IndexList Indices;
VertexList Vertices;
};


The function prototype for the extrusion method needs three arguments: the source StickFigure (lines), the destination StickFigure (triangles), and the desired line width. See Example 6-18 and refer back to Figure 6-11 to visualize the six extrusion vectors (N, S, NE, NW, SW, SE).

Example 6-18. Line extrusion algorithm
void ExtrudeLines(const StickFigure& lines, StickFigure& triangles, float width)
{
IndexList::iterator sourceIndex = lines.Indices.begin();
VertexList::iterator destVertex = triangles.Vertices.begin();
while (sourceIndex != lines.Indices.end()) {

vec3 a = lines.Vertices[lines.Indices[*sourceIndex++]].Position;
vec3 b = lines.Vertices[lines.Indices[*sourceIndex++]].Position;
vec3 e = (b - a).Normalized() * width;

vec3 N = vec3(-e.y, e.x, 0);
vec3 S = -N;
vec3 NE = N + e;
vec3 NW = N - e;
vec3 SW = -NE;
vec3 SE = -NW;

destVertex++->Position = a + SW;
destVertex++->Position = a + NW;
destVertex++->Position = a + S;
destVertex++->Position = a + N;
destVertex++->Position = b + S;
destVertex++->Position = b + N;
destVertex++->Position = b + SE;
destVertex++->Position = b + NE;
}
}


At this point, we’ve computed the positions of the extruded triangles, but we still haven’t provided texture coordinates for the triangles, nor the contents of the index buffer. Note that the animated figure can change its vertex positions at every frame, but the number of lines stays the same. This means we can generate the index list only once; there’s no need to recompute it at every frame. The same goes for the texture coordinates. Let’s declare a couple functions for these start-of-day tasks:

void GenerateTriangleIndices(size_t lineCount, IndexList& triangles);
void GenerateTriangleTexCoords(size_t lineCount, VertexList& triangles);


Flip back to Figure 6-9, and note the number of triangles and vertices. Every line primitive extrudes into six triangles composed from eight vertices. Since every triangle requires three indices, the number of indices in the new index buffer is lineCount*18. This is different from the number of vertices, which is only lineCount*8. See Example 6-19.

Example 6-19. Line extrusion initialization methods
void GenerateTriangleIndices(size_t lineCount, IndexList& triangles)
{
triangles.resize(lineCount * 18);
IndexList::iterator index = triangles.begin();
for (GLushort v = 0; index != triangles.end(); v += 8) {
*index++ = 0 + v; *index++ = 1 + v; *index++ = 2 + v;
*index++ = 2 + v; *index++ = 1 + v; *index++ = 3 + v;
*index++ = 2 + v; *index++ = 3 + v; *index++ = 4 + v;
*index++ = 4 + v; *index++ = 3 + v; *index++ = 5 + v;
*index++ = 4 + v; *index++ = 5 + v; *index++ = 6 + v;
*index++ = 6 + v; *index++ = 5 + v; *index++ = 7 + v;
}
}

void GenerateTriangleTexCoords(size_t lineCount, VertexList& triangles)
{
triangles.resize(lineCount * 8);
VertexList::iterator vertex = triangles.begin();
while (vertex != triangles.end()) {
vertex++->TexCoord = vec2(0, 0);
vertex++->TexCoord = vec2(0, 1);
vertex++->TexCoord = vec2(0.5, 0);
vertex++->TexCoord = vec2(0.5, 1);
vertex++->TexCoord = vec2(0.5, 0);
vertex++->TexCoord = vec2(0.5, 1);
vertex++->TexCoord = vec2(1, 0);
vertex++->TexCoord = vec2(1, 1);
}
}


Et voilà…you now know how to render antialiased lines on a device that doesn’t support antialiased lines! To see this in action, check out the AaLines sample from this book’s example code.

## Holodeck Sample

In this chapter’s introduction, we promised to present a poor man’s augmented reality app. As a starting point, we’ll create a 3D environment that includes the aforementioned geodesic dome with antialiased borders. We’ll also render a mossy ground plane and some moving clouds in the background. Later we’ll replace the clouds with a live camera image. Another interesting aspect to this sample is that it’s designed for landscape mode; see Figure 6-12.

For rendering the AA lines in the dome, let’s use a different trick than the one presented in the previous section. Rather than a filling a texture with a circle, let’s fill it with a triangle, as shown in Figure 6-13. By choosing texture coordinates in the right places (see the hollow circles in the figure), we’ll be creating a thick border at every triangle.

For controlling the camera, the app should use the compass and accelerometer APIs to truly qualify as an augmented reality app. However, initially let’s just show four buttons in a HUD: touching any button will cause the environment to “scroll.” Horizontal buttons control azimuth (angle from north); vertical buttons control altitude (angle above horizon). These terms may be familiar to you if you’re an astronomy buff.

Later we’ll replace the azimuth/altitude buttons with the compass and accelerometer APIs. The benefit of this approach is that we can easily provide a fallback option if the app discovers that the compass or accelerometer APIs are not available. This allows us to gracefully handle three scenarios:

iPhone Simulator

Show buttons for both azimuth and altitude.

First- and second-generation iPhones

Show buttons for azimuth; use the accelerometer for altitude.

Third-generation iPhones

Hide all buttons; use the accelerometer for altitude and the compass for azimuth.

In honor of my favorite TV show, the name of this sample is Holodeck. Without further ado, let’s begin!

### Application Skeleton

The basic skeleton for the Holodeck sample is much like every other sample we’ve presented since Chapter 3. The main difference is that we forgo the creation of an IApplicationEngine interface and instead place the application logic directly within the GLView class. There’s very little logic required for this app anyway; most of the heavy footwork is done in the rendering engine. Skipping the application layer makes life easier when we add support for the accelerometer, compass, and camera APIs.

Another difference lies in how we handle the dome geometry. Rather than loading in the vertices from an OBJ file or generating them at runtime, a Python script generates a C++ header file with the dome data, as shown in Example 6-20; you can download the full listing, along with the Holodeck project, from this book’s website. This is perhaps the simplest possible way to load geometry into an OpenGL application, and some modeling tools can actually export their data as a C/C++ header file!

Example 6-20. GeodesicDome.h
const int DomeFaceCount = 2782;
const int DomeVertexCount = DomeFaceCount * 3;
const float DomeVertices[DomeVertexCount * 5] = {
-0.819207, 0.040640, 0.572056,
0.000000, 1.000000,

...

0.859848, -0.065758, 0.506298,
1.000000, 1.000000,
};


Figure 6-14 shows the overall structure of the Holodeck project.

Note that this app has quite a few textures compared to our previous samples: six PNG files and two compressed PVRTC files. You can also see from the screenshot that we’ve added a new property to Info.plist called UIInterfaceOrientation. Recall that this is a landscape-only app; if you don’t set this property, you’ll have to manually rotate the virtual iPhone every time you test it in the simulator.

Interfaces.hpp is much the same as in our other sample apps, except that the rendering engine interface is somewhat unique; see Example 6-21.

Example 6-21. Interfaces.hpp for Holodeck
...

enum ButtonFlags {
ButtonFlagsShowHorizontal = 1 << 0,
ButtonFlagsShowVertical = 1 << 1,
ButtonFlagsPressingUp = 1 << 2,
ButtonFlagsPressingDown = 1 << 3,
ButtonFlagsPressingLeft = 1 << 4,
ButtonFlagsPressingRight = 1 << 5,
};

struct IRenderingEngine {
virtual void Initialize() = 0;
virtual void Render(float theta, float phi,
virtual ~IRenderingEngine() {}
};

...


The new Render method takes three parameters:

float theta

Azimuth in degrees. This is the horizontal angle off east.

float phi

Altitude in degrees. This is the vertical angle off the horizon.

Bit mask of flags for the HUD.

The idea behind the buttons mask is that the Objective-C code (GLView.mm) can determine the capabilities of the device and whether a button is being pressed, so it sends this information to the rendering engine as a set of flags.

### Rendering the Dome, Clouds, and Text

For now let’s ignore the buttons and focus on rendering the basic elements of the 3D scene. See Example 6-22 for the rendering engine declaration and related types. Utility methods that carry over from previous samples, such as CreateTexture, are replaced with ellipses for brevity.

Example 6-22. RenderingEngine declaration for Holodeck
struct Drawable {
GLuint VertexBuffer;
GLuint IndexBuffer;
int IndexCount;
int VertexCount;
};

struct Drawables {
Drawable GeodesicDome;
Drawable SkySphere;
};

struct Textures {
GLuint Sky;
GLuint Floor;
GLuint Button;
GLuint Triangle;
GLuint North;
GLuint South;
GLuint East;
GLuint West;
};

struct Renderbuffers {
GLuint Color;
GLuint Depth;
};

class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize();
void Render(float theta, float phi, ButtonMask buttonFlags) const;
private:
void RenderText(GLuint texture, float theta, float scale) const;
Drawable CreateDrawable(const float* vertices, int vertexCount);
// ...
Drawables m_drawables;
Textures m_textures;
Renderbuffers m_renderbuffers;
IResourceManager* m_resourceManager;
};


Note that Example 6-22 declares two new private methods: RenderText for drawing compass direction labels and a new CreateDrawable method for creating the geodesic dome. Even though it declares eight different texture objects (which could be combined into a texture atlas; see Chapter 7), it declares only three VBOs. The Quad VBO is re-used for the buttons, the floor, and the floating text.

Example 6-23 is fairly straightforward. It first creates the VBOs and texture objects and then initializes various OpenGL state.

Example 6-23. RenderingEngine initialization for Holodeck
#include "../Models/GeodesicDome.h"

...

void RenderingEngine::Initialize()
{
// Create vertex buffer objects.
m_drawables.GeodesicDome =
CreateDrawable(DomeVertices, DomeVertexCount);
m_drawables.SkySphere = CreateDrawable(Sphere(1));

m_textures.Floor = CreateTexture("Moss.pvr");
m_textures.Sky = CreateTexture("Sky.pvr");
m_textures.Button = CreateTexture("Button.png");
m_textures.Triangle = CreateTexture("Triangle.png");
m_textures.North = CreateTexture("North.png");
m_textures.South = CreateTexture("South.png");
m_textures.East = CreateTexture("East.png");
m_textures.West = CreateTexture("West.png");

// Extract width and height from the color buffer.
int width, height;
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_WIDTH_OES, &width);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_HEIGHT_OES, &height);
glViewport(0, 0, width, height);

// Create a depth buffer that has the same size as the color buffer.
glGenRenderbuffersOES(1, &m_renderbuffers.Depth);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.Depth);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES,
GL_DEPTH_COMPONENT16_OES, width, height);

// Create the framebuffer object.
GLuint framebuffer;
glGenFramebuffersOES(1, &framebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, framebuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.Color);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_DEPTH_ATTACHMENT_OES,
GL_RENDERBUFFER_OES,
m_renderbuffers.Depth);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_renderbuffers.Color);

// Set up various GL state.
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glEnable(GL_TEXTURE_2D);
glEnable(GL_DEPTH_TEST);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

// Set the model-view transform.
glMatrixMode(GL_MODELVIEW);
glRotatef(90, 0, 0, 1);

// Set the projection transform.
float h = 4.0f * height / width;
glMatrixMode(GL_PROJECTION);
glFrustumf(-2, 2, -h / 2, h / 2, 5, 200);
glMatrixMode(GL_MODELVIEW);
}

Drawable RenderingEngine::CreateDrawable(const float* vertices,
int vertexCount)
{
// Each vertex has XYZ and ST, for a total of five floats.
const int FloatsPerVertex = 5;

// Create the VBO for the vertices.
GLuint vertexBuffer;
glGenBuffers(1, &vertexBuffer);
glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
glBufferData(GL_ARRAY_BUFFER,
vertexCount * FloatsPerVertex * sizeof(float),
vertices,
GL_STATIC_DRAW);

// Fill in the description structure and return it.
Drawable drawable = {0};
drawable.VertexBuffer = vertexBuffer;
drawable.VertexCount = vertexCount;
return drawable;
}


Let’s finally take a look at the all-important Render method; see Example 6-24.

Example 6-24. Render method for Holodeck
void RenderingEngine::Render(float theta, float phi,
{
static float frameCounter = 0;
frameCounter++;

glPushMatrix();

glRotatef(phi, 1, 0, 0);
glRotatef(theta, 0, 1, 0);

glClear(GL_DEPTH_BUFFER_BIT);

glPushMatrix();
glScalef(100, 100, 100);
glRotatef(frameCounter * 2, 0, 1, 0);
glBindTexture(GL_TEXTURE_2D, m_textures.Sky);
RenderDrawable(m_drawables.SkySphere);
glPopMatrix();

glEnable(GL_BLEND);
glBindTexture(GL_TEXTURE_2D, m_textures.Triangle);
glPushMatrix();
glTranslatef(0, 10, 0);
glScalef(90, 90, 90);
glColor4f(1, 1, 1, 0.75f);
RenderDrawable(m_drawables.GeodesicDome);
glColor4f(1, 1, 1, 1);
glPopMatrix();

float textScale = 1.0 / 10.0 + sin(frameCounter / 10.0f) / 150.0;

RenderText(m_textures.East, 0, textScale);
RenderText(m_textures.West, 180, textScale);
RenderText(m_textures.South, 90, textScale);
RenderText(m_textures.North, -90, textScale);
glDisable(GL_BLEND);

glTranslatef(0, 10, -10);
glRotatef(90, 1, 0, 0);
glScalef(4, 4, 4);
glMatrixMode(GL_TEXTURE);
glScalef(4, 4, 1);
glBindTexture(GL_TEXTURE_2D, m_textures.Floor);
glMatrixMode(GL_MODELVIEW);
glPopMatrix();

if (buttons) {
...
}
}

 Use a static variable to keep a frame count for animation. I don’t recommend this approach in production code (normally you’d use a delta-time value), but this is fine for an example. Rotate theta degrees (azimuth) around the y-axis and phi degrees (altitude) around the x-axis. We’re clearing depth only; there’s no need to clear color since we’re drawing a sky sphere. Render the sky sphere. Render the geodesic dome with blending enabled. Create an animated variable called textScale for the pulse effect, and then pass it in to the RenderText method. Draw the mossy ground plane. Render the buttons only if the buttons mask is nonzero. We’ll cover button rendering shortly.

The RenderText method is fairly straightforward; see Example 6-25. Some glScalef trickery is used to stretch out the quad and flip it around.

Example 6-25. RenderText method for Holodeck
void RenderingEngine::RenderText(GLuint texture, float theta,
float scale) const
{
glBindTexture(GL_TEXTURE_2D, texture);
glPushMatrix();
glRotatef(theta, 0, 1, 0);
glTranslatef(0, -2, -30);
glScalef(-2 * scale, -scale, scale);
glPopMatrix();
}


Most applications that need to render a HUD take the following approach when rendering a single frame of animation:

1. Issue a glClear.

2. Set up the model-view and projection matrices for the 3D scene.

3. Render the 3D scene.

4. Disable depth testing, and enable blending.

5. Set up the model-view and projection matrices for 2D rendering.

6. Render the HUD.

Always remember to completely reset your transforms at the beginning of the render routine; otherwise, you’ll apply transformations that are left over from the previous frame. For example, calling glFrustum alone simply multiplies the current matrix, so you might need to issue a glLoadIdentity immediately before calling glFrustum.

Let’s go ahead and modify the Render method to render buttons; replace the ellipses in Example 6-24 with the code in Example 6-26.

Example 6-26. Adding buttons to Holodeck
glEnable(GL_BLEND);
glDisable(GL_DEPTH_TEST);
glBindTexture(GL_TEXTURE_2D, m_textures.Button);
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glOrthof(-160, 160, -240, 240, 0, 1);

if (buttons & ButtonFlagsShowHorizontal) {
glMatrixMode(GL_MODELVIEW);
glTranslatef(200, 0, 0);
SetButtonAlpha(buttons, ButtonFlagsPressingLeft);
glTranslatef(-400, 0, 0);
glMatrixMode(GL_TEXTURE);
glRotatef(180, 0, 0, 1);
SetButtonAlpha(buttons, ButtonFlagsPressingRight);
glRotatef(-180, 0, 0, 1);
glMatrixMode(GL_MODELVIEW);
glTranslatef(200, 0, 0);
}

if (buttons & ButtonFlagsShowVertical) {
glMatrixMode(GL_MODELVIEW);
glTranslatef(0, 125, 0);
glMatrixMode(GL_TEXTURE);
glRotatef(90, 0, 0, 1);
SetButtonAlpha(buttons, ButtonFlagsPressingUp);
glMatrixMode(GL_MODELVIEW);
glTranslatef(0, -250, 0);
glMatrixMode(GL_TEXTURE);
glRotatef(180, 0, 0, 1);
SetButtonAlpha(buttons, ButtonFlagsPressingDown);
glRotatef(90, 0, 0, 1);
glMatrixMode(GL_MODELVIEW);
glTranslatef(0, 125, 0);
}

glColor4f(1, 1, 1, 1);
glMatrixMode(GL_PROJECTION);
glPopMatrix();
glMatrixMode(GL_MODELVIEW);
glEnable(GL_DEPTH_TEST);
glDisable(GL_BLEND);

Note that Example 6-26 contains quite a few transform operations; while this is fine for teaching purposes, in a production environment I recommend including all four buttons in a single VBO. You’d still need four separate draw calls, however, since the currently pressed button has a unique alpha value.

In fact, making this optimization would be an interesting project: create a single VBO that contains all four pretransformed buttons, and then render it with four separate draw calls. Don’t forget that the second argument to glDrawArrays can be nonzero!

The SetButtonAlpha method sets alpha to one if the button is being pressed; otherwise, it makes the button semitransparent:

void RenderingEngine::SetButtonAlpha(ButtonMask buttonFlags,
ButtonFlags flag) const
{
float alpha = (buttonFlags & flag) ? 1.0 : 0.75;
glColor4f(1, 1, 1, alpha);
}


Next let’s go over the code in GLView.mm that detects button presses and maintains the azimuth/altitude angles. See Example 6-27 for the GLView class declaration and Example 6-28 for the interesting potions of the class implementation.

Example 6-27. GLView.h for Holodeck
#import "Interfaces.hpp"
#import <UIKit/UIKit.h>
#import <QuartzCore/QuartzCore.h>
#import <CoreLocation/CoreLocation.h>

@interface GLView : UIView {
@private
IRenderingEngine* m_renderingEngine;
IResourceManager* m_resourceManager;
EAGLContext* m_context;
bool m_paused;
float m_theta;
float m_phi;
vec2 m_velocity;
float m_timestamp;
}

@end

Example 6-28. GLView.mm for Holodeck
...

- (id) initWithFrame: (CGRect) frame
{
m_paused = false;
m_theta = 0;
m_phi = 0;
m_velocity = vec2(0, 0);
m_visibleButtons = ButtonFlagsShowHorizontal | ButtonFlagsShowVertical;

if (self = [super initWithFrame:frame]) {
CAEAGLLayer* eaglLayer = (CAEAGLLayer*) self.layer;
eaglLayer.opaque = YES;

EAGLRenderingAPI api = kEAGLRenderingAPIOpenGLES1;
m_context = [[EAGLContext alloc] initWithAPI:api];

if (!m_context || ![EAGLContext setCurrentContext:m_context]) {
[self release];
return nil;
}

m_resourceManager = CreateResourceManager();

NSLog(@"Using OpenGL ES 1.1");
m_renderingEngine = CreateRenderingEngine(m_resourceManager);

[m_context
renderbufferStorage:GL_RENDERBUFFER
fromDrawable: eaglLayer];

m_timestamp = CACurrentMediaTime();

m_renderingEngine->Initialize();
[self drawView:nil];

selector:@selector(drawView:)];

forMode:NSDefaultRunLoopMode];
}
return self;
}

{
if (m_paused)
return;

const float speed = 30;
float elapsedSeconds = displayLink.timestamp - m_timestamp;
m_theta -= speed * elapsedSeconds * m_velocity.x;
m_phi += speed * elapsedSeconds * m_velocity.y;
}

if (m_velocity.x < 0) buttonFlags |= ButtonFlagsPressingLeft;
if (m_velocity.x > 0) buttonFlags |= ButtonFlagsPressingRight;
if (m_velocity.y < 0) buttonFlags |= ButtonFlagsPressingUp;
if (m_velocity.y > 0) buttonFlags |= ButtonFlagsPressingDown;

m_renderingEngine->Render(m_theta, m_phi, buttonFlags);
[m_context presentRenderbuffer:GL_RENDERBUFFER];
}

bool buttonHit(CGPoint location, int x, int y)
{
float extent = 32;
return (location.x > x - extent && location.x < x + extent &&
location.y > y - extent && location.y < y + extent);
}

- (void) touchesBegan: (NSSet*) touches withEvent: (UIEvent*) event
{
UITouch* touch = [touches anyObject];
CGPoint location  = [touch locationInView: self];
float delta = 1;

if (m_visibleButtons & ButtonFlagsShowVertical) {
if (buttonHit(location, 35, 240))
m_velocity.y = -delta;
else if (buttonHit(location, 285, 240))
m_velocity.y = delta;
}

if (m_visibleButtons & ButtonFlagsShowHorizontal) {
if (buttonHit(location, 160, 40))
m_velocity.x = -delta;
else if (buttonHit(location, 160, 440))
m_velocity.x = delta;
}
}

- (void) touchesEnded: (NSSet*) touches withEvent: (UIEvent*) event
{
m_velocity = vec2(0, 0);
}

 For now, we’re hardcoding both button visibility flags to true. We’ll make this dynamic after adding compass and accelerometer support. The theta and phi angles are updated according to the current velocity vector and delta time. Right before passing in the button mask to the Render method, take a look at the velocity vector to decide which buttons are being pressed. Simple utility function to detect whether a given point (location) is within the bounds of a button centered at (x, y). Note that we’re allowing the intrusion of a vanilla C function into an Objective-C file. To make things simple, the velocity vector is set up in response to a “finger down” event and reset to zero in response to a “finger up” event. Since we don’t need the ability for several buttons to be pressed simultaneously, this is good enough.

At this point, you now have a complete app that lets you look around inside a (rather boring) virtual world, but it’s still a far cry from augmented reality!

### Replacing Buttons with Orientation Sensors

The next step is carefully integrating support for the compass and accelerometer APIs. I say “carefully” because we’d like to provide a graceful runtime fallback if the device (or simulator) does not have a magnetometer or accelerometer.

We’ll be using the accelerometer to obtain the gravity vector, which in turn enables us to compute the phi angle (that’s “altitude” for you astronomers) but not the theta angle (azimuth). Conversely, the compass API can be used to compute theta but not phi. You’ll see how this works in the following sections.

Using the low-level accelerometer API directly is ill advised; the signal includes quite a bit of noise, and unless your app is somehow related to The Blair Witch Project, you probably don’t want your camera shaking around like a shivering chihuahua.

Discussing a robust and adaptive low-pass filter implementation is beyond the scope of this book, but thankfully Apple includes some example code for this. Search for the AccelerometerGraph sample on the iPhone developer site (http://developer.apple.com/iphone) and download it. Look inside for two key files, and copy them to your project folder: AccelerometerFilter.h and AccelerometerFilter.m.

You can also refer to “Stabilizing the counter with a low-pass filter” for an example implementation of a simple low-pass filter.

After adding the filter code to your Xcode project, open up GLView.h, and add the three code snippets that are highlighted in bold in Example 6-29.

Example 6-29. Adding accelerometer support to GLView.h
#import "Interfaces.hpp"
#import "AccelerometerFilter.h"
#import <UIKit/UIKit.h>
#import <QuartzCore/QuartzCore.h>

@interface GLView : UIView <UIAccelerometerDelegate> {
@private
IRenderingEngine* m_renderingEngine;
IResourceManager* m_resourceManager;
EAGLContext* m_context;
AccelerometerFilter* m_filter;
...
}

@end


Next, open GLView.mm, and add the lines shown in bold in Example 6-30. You might grimace at the sight of the #if block, but it’s a necessary evil because the iPhone Simulator pretends to support the accelerometer APIs by sending the application fictitious values (without giving the user much control over those values). Since the fake accelerometer won’t do us much good, we turn it off when building for the simulator.

An Egyptian software company called vimov produces a compelling tool called iSimulate that can simulate the accelerometer and other device sensors. Check it out at http://www.vimov.com/isimulate.

Example 6-30. Adding accelerometer support to initWithFrame
- (id) initWithFrame: (CGRect) frame
{
m_paused = false;
m_theta = 0;
m_phi = 0;
m_velocity = vec2(0, 0);
m_visibleButtons = 0;

if (self = [super initWithFrame:frame]) {
CAEAGLLayer* eaglLayer = (CAEAGLLayer*) self.layer;
eaglLayer.opaque = YES;

EAGLRenderingAPI api = kEAGLRenderingAPIOpenGLES1;
m_context = [[EAGLContext alloc] initWithAPI:api];

if (!m_context || ![EAGLContext setCurrentContext:m_context]) {
[self release];
return nil;
}

m_resourceManager = CreateResourceManager();

NSLog(@"Using OpenGL ES 1.1");
m_renderingEngine = CreateRenderingEngine(m_resourceManager);

#if TARGET_IPHONE_SIMULATOR
BOOL compassSupported = NO;
BOOL accelSupported = NO;
#else
BOOL compassSupported = NO; // (We'll add compass support shortly.)
BOOL accelSupported = YES;
#endif

if (compassSupported) {
NSLog(@"Compass is supported.");
} else {
NSLog(@"Compass is NOT supported.");
m_visibleButtons |= ButtonFlagsShowHorizontal;
}

if (accelSupported) {
NSLog(@"Accelerometer is supported.");
float updateFrequency = 60.0f;
m_filter =
[[LowpassFilter alloc] initWithSampleRate:updateFrequency
cutoffFrequency:5.0];

[[UIAccelerometer sharedAccelerometer]
setUpdateInterval:1.0 / updateFrequency];
[[UIAccelerometer sharedAccelerometer] setDelegate:self];
} else {
NSLog(@"Accelerometer is NOT supported.");
m_visibleButtons |= ButtonFlagsShowVertical;
}

[m_context
renderbufferStorage:GL_RENDERBUFFER
fromDrawable: eaglLayer];

m_timestamp = CACurrentMediaTime();

m_renderingEngine->Initialize();
[self drawView:nil];

selector:@selector(drawView:)];

forMode:NSDefaultRunLoopMode];
}
return self;
}


Since GLView sets itself as the accelerometer delegate, it needs to implement a response handler. See Example 6-31.

Example 6-31. Accelerometer response handler
- (void) accelerometer: (UIAccelerometer*) accelerometer
didAccelerate: (UIAcceleration*) acceleration
{
float x = m_filter.x;
float z = m_filter.z;
m_phi = atan2(z, -x) * 180.0f / Pi;
}


You might not be familiar with the atan2 function, which takes the arctangent of the its first argument divided by the its second argument (see Equation 6-1). Why not use the plain old single-argument atan function and do the division yourself? You don’t because atan2 is smarter; it uses the signs of its arguments to determine which quadrant the angle is in. Plus, it allows the second argument to be zero without throwing a divide-by-zero exception.

An even more rarely encountered math function is hypot. When used together, atan2 and hypot can convert any 2D Cartesian coordinate into a polar coordinate.

Equation 6-1. Phi as a function of acceleration

Equation 6-1 shows how we compute phi from the accelerometer’s input values. To understand it, you first need to realize that we’re using the accelerometer as a way of measuring the direction of gravity. It’s a common misconception that the accelerometer measures speed, but you know better by now! The accelerometer API returns a 3D acceleration vector according to the axes depicted in Figure 6-15.

When you hold the device in landscape mode, there’s no gravity along the y-axis (assuming you’re not slothfully laying on the sofa and turned to one side). So, the gravity vector is composed of X and Z only—see Figure 6-16.

The direction of gravity can’t tell you which direction you’re facing; that’s where the compass support in third-generation devices comes in. To begin, open GLView.h, and add the bold lines in Example 6-32.

Example 6-32. Adding compass support to GLView.h
#import "Interfaces.hpp"
#import "AccelerometerFilter.h"
#import <UIKit/UIKit.h>
#import <QuartzCore/QuartzCore.h>
#import <CoreLocation/CoreLocation.h>

@interface GLView : UIView <CLLocationManagerDelegate,
UIAccelerometerDelegate> {
@private
IRenderingEngine* m_renderingEngine;
IResourceManager* m_resourceManager;
EAGLContext* m_context;
CLLocationManager* m_locationManager;
AccelerometerFilter* m_filter;
...
}

@end


The Core Location API is an umbrella for both GPS and compass functionality, but we’ll be using only the compass functionality in our demo. Next we need to create an instance of CLLocationManger somewhere in GLview.mm; see Example 6-33.

Example 6-33. Adding compass support to initWithFrame
- (id) initWithFrame: (CGRect) frame
{
...

if (self = [super initWithFrame:frame]) {

...

m_locationManager = [[CLLocationManager alloc] init];

#if TARGET_IPHONE_SIMULATOR
BOOL compassSupported = NO;
BOOL accelSupported = NO;
#else
BOOL accelSupported = YES;
#endif

if (compassSupported) {
NSLog(@"Compass is supported.");
m_locationManager.delegate = self;
} else {
NSLog(@"Compass is NOT supported.");
m_visibleButtons |= ButtonFlagsShowHorizontal;
}

...
}
return self;
}


Similar to how it handles the accelerometer feedback, GLView sets itself as the compass delegate, so it needs to implement a response handler. See Example 6-31. Unlike the accelerometer, any noise in the compass reading is already eliminated, so there’s no need for handling the low-pass filter yourself. The compass API is embarrassingly simple; it simply returns an angle in degrees, where 0 is north, 90 is east, and so on. See Example 6-34 for the compass response handler.

Example 6-34. Compass response handler
- (void) locationManager: (CLLocationManager*) manager
{
m_theta = (float) -degrees;
}


The only decision you have to make when writing a compass handler is whether to use magneticHeading or trueHeading. The former returns magnetic north, which isn’t quite the same as geographic north. To determine the true direction of the geographic north pole, the device needs to know where it’s located on the planet, which requires usage of the GPS. Since our app is looking around a virtual world, it doesn’t matter which heading to use. I chose to use magneticHeading because it allows us to avoid enabling GPS updates in the location manager object. This simplifies the code and may even improve power consumption.

### Overlaying with a Live Camera Image

To make this a true augmented reality app, we need to bring the camera into play. If a camera isn’t available (as in the simulator), then the app can simply fall back to the “scrolling clouds” background.

The first step is adding another protocol to the GLView class—actually we need two new protocols! Add the bold lines in Example 6-35, noting the new data fields as well (m_viewController and m_cameraSupported).

Example 6-35. Adding camera support to GLView.h
#import "Interfaces.hpp"
#import "AccelerometerFilter.h"
#import <UIKit/UIKit.h>
#import <QuartzCore/QuartzCore.h>
#import <CoreLocation/CoreLocation.h>

@interface GLView : UIView <UIImagePickerControllerDelegate,
CLLocationManagerDelegate,
UIAccelerometerDelegate> {
@private
IRenderingEngine* m_renderingEngine;
IResourceManager* m_resourceManager;
EAGLContext* m_context;
CLLocationManager* m_locationManager;
AccelerometerFilter* m_filter;
UIViewController* m_viewController;
bool m_cameraSupported;
...
}

@end


Next we need to enhance the initWithFrame and drawView methods. See Example 6-36. Until now, every sample in this book has set the opaque property in the EAGL layer to YES. In this sample, we decide its value at runtime; if a camera is available, don’t make the surface opaque to allow the image “underlay” to show through.

Example 6-36. Adding camera support to GLView.mm
- (id) initWithFrame: (CGRect) frame
{
...

if (self = [super initWithFrame:frame]) {

m_cameraSupported = [UIImagePickerController isSourceTypeAvailable:
UIImagePickerControllerSourceTypeCamera];

CAEAGLLayer* eaglLayer = (CAEAGLLayer*) self.layer;
eaglLayer.opaque = !m_cameraSupported;
if (m_cameraSupported)
NSLog(@"Camera is supported.");
else
NSLog(@"Camera is NOT supported.");

...

#if TARGET_IPHONE_SIMULATOR
BOOL compassSupported = NO;
BOOL accelSupported = NO;
#else
BOOL accelSupported = YES;
#endif

m_viewController = 0;

...

m_timestamp = CACurrentMediaTime();

bool opaqueBackground = !m_cameraSupported;
m_renderingEngine->Initialize(opaqueBackground);

// Delete the line [self drawView:nil];

selector:@selector(drawView:)];

...
}
return self;
}

{
if (m_cameraSupported && m_viewController == 0)
[self createCameraController];

if (m_paused)
return;

...

m_renderingEngine->Render(m_theta, m_phi, buttonFlags);
[m_context presentRenderbuffer:GL_RENDERBUFFER];
}



Next we need to implement the createCameraController method that was called from drawView. This is an example of lazy instantiation; we don’t create the camera controller until we actually need it. Example 6-37 shows the method, and a detailed explanation follows the listing. (The createCameraController method needs to be defined before the drawView method to avoid a compiler warning.)

Example 6-37. Creating the camera view controller
- (void) createCameraController
{
UIImagePickerController* imagePicker =
[[UIImagePickerController alloc] init];
imagePicker.delegate = self;
imagePicker.toolbarHidden = YES;
imagePicker.sourceType = UIImagePickerControllerSourceTypeCamera;
imagePicker.showsCameraControls = NO;
imagePicker.cameraOverlayView = self;

// The 54 pixel wide empty spot is filled in by scaling the image.
// The camera view's height gets stretched from 426 pixels to 480.

float bandWidth = 54;
float screenHeight = 480;
float zoomFactor = screenHeight / (screenHeight - bandWidth);

CGAffineTransform pickerTransform =
CGAffineTransformMakeScale(zoomFactor, zoomFactor);
imagePicker.cameraViewTransform = pickerTransform;

m_viewController = [[UIViewController alloc] init];
m_viewController.view = self;
[m_viewController presentModalViewController:imagePicker animated:NO];
}

 Set the image picker’s delegate to the GLView class. Since we aren’t using the camera to capture still images, this isn’t strictly necessary, but it’s still a good practice. Hide the navigation bar. Again, we aren’t using the camera for image capture, so there’s no need for this UI getting in the way. Ditto with the toolbar. Set the source type of the image picker to the camera. You might recall this step from the camera texture sample in the previous chapter. Hide the camera control UI. Again, we’re using the camera only as a backdrop, so any UI would just get in the way. Set the camera overlay view to the GLView class to allow the OpenGL content to be rendered. The UI that we’re hiding would normally leave an annoying gap on the bottom of the screen. By applying a scale transform, we can fill in the gap. Maintaining the correct aspect ratio causes a portion of the image to be cropped, but it’s not noticeable in the final app. Finally, present the view controller to make the camera image show up.

Since we’re using the camera API in a way that’s quite different from how Apple intended, we had to jump through a few hoops: hiding the UI, stretching the image, and implementing a protocol that never really gets used. This may seem a bit hacky, but ideally Apple will improve the camera API in the future to simplify the development of augmented reality applications.

You may’ve noticed in Example 6-36 that the view class is now passing in a boolean to the rendering engine’s Initialize method; this tells it whether the background should contain clouds as before or whether it should be cleared to allow the camera underlay to show through. You must modify the declaration of Initialize in Interfaces.cpp accordingly. Next, the only remaining changes are shown in Example 6-38.

Example 6-38. RenderingEngine modifications to support the camera “underlay”
...

class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize(bool opaqueBackground);
void Render(float theta, float phi, ButtonMask buttons) const;
private:
...
bool m_opaqueBackground;
};

void RenderingEngine::Initialize(bool opaqueBackground)
{
m_opaqueBackground = opaqueBackground;

...
}

void RenderingEngine::Render(float theta, float phi, ButtonMask buttons) const
{
static float frameCounter = 0;
frameCounter++;

glPushMatrix();

glRotatef(phi, 1, 0, 0);
glRotatef(theta, 0, 1, 0);

if (m_opaqueBackground) {
glClear(GL_DEPTH_BUFFER_BIT);

glPushMatrix();
glScalef(100, 100, 100);
glRotatef(frameCounter * 2, 0, 1, 0);
glBindTexture(GL_TEXTURE_2D, m_textures.Sky);
RenderDrawable(m_drawables.SkySphere);
glPopMatrix();
} else {
glClearColor(0, 0, 0, 0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
}

...
}



Note that the alpha value of the clear color is zero; this allows the underlay to show through. Also note that the color buffer is cleared only if there’s no sky sphere. Experienced OpenGL programmers make little optimizations like this as a matter of habit.

That’s it for the Holodeck sample! See Figure 6-17 for a depiction of the app as it now stands.

## Wrapping Up

In this chapter we learned how to put FBOs to good use for the first time. We learned how to achieve anti-aliasing in sneaky ways, how to layer a scene by mixing 2D content with 3D content, and how to use the iPhone’s orientation sensors in tandem with OpenGL.

We explored the concept of a 2D HUD in the Holodeck sample, but we largely glossed over the subject of text. Supplying ready-made textures of complete words (as we did for Holodeck) can be a bit cumbersome; often an application needs to render large amounts of dynamic text together with a 3D scene. Since text is something that OpenGL can’t really handle on its own (and justifiably so), it deserves more attention. This brings us to the next chapter.