Reality Composer

Overview

"Reality Composer" is an iOS and macOS app that allows users to "easily prototype and produce AR experiences directly with no prior 3D experience". The macOS version is included with Xcode and cannot be downloaded separately.

The app can import virtual objects from existing 3D models, add animations to them, and add audio that can be triggered in many ways. The app can also "record sensor and camera data in the actual location you are building it for, and then replay it on your iOS device".

Non-Apple alternatives to Reality Composer include:

To launch the "Reality Composer" app in macOS, launch Xcode and select Xcode ... Open Developer Tool .. Reality Composer.

Resources

Florian Schweizer (@FloWritesCode) created the following excellent YouTube video on Reality Composer:

Terminology

Creating a Project

To create a new project:

To rename a project:

Creating a Scene

A project begins with one default scene.

To create another scene:

To select a scene:

To rename a scene:

To delete a scene:

To duplicate a scene:

To enable switching from one scene to another when the project is running, add a "Change Scene" action to a behavior and select a destination scene. For example, this can be an action in a "Tap" behavior so tapping an object triggers a scene change. Perhaps this can be used to simulate exiting a room and entering another.

To transform the view of a scene:

Anchors

An anchor ties objects in a scene to a scene location such as a coordinate relative to an iPhone location when the app launches.

The supported anchor types include:

When an anchor is created in code, it must be added to a scene with scene.addAnchor(myAnchor).

Creating Objects

To add an object to the scene:

Configuring an Object

To toggle display of the Inspector panel on the right side, tap the properties button in the upper-right (cube in a gear in a circle).

To change the position, rotation, and scale of an object:

To change the "look" of an object:

Basic shapes can also select a "Material" and a "Material Color". Material options include "Glossy Paint", "Matte Paint", "Plastic", "Car Paint", "Aluminum", "Brass", "Bronze", "Gold", "Steel", "Rubber", and "Terracotta".

Objects can be grouped so they can be transformed as a single unit. To select all objects in the scene, tap outside all of them and tap "Select All". To select a subset of the objects, hold down on one object and tap all the others. After selecting the objects to be grouped, tap any one of them to get a popup of options, and tap "Group". To ungroup the objects in a group, tap the group to selected it, tap again to get a popup of options, and tap "Ungroup".

Physics

Objects can optionally participate in physics. When participating objects are off the ground and their scene is played, gravity causes them to fall toward the ground.

To cause an object to participate in physics:

Duplicating an Object

To duplicate an object, tap the object, tap it again to get a popup of options, and tap "Duplicate". By default this does not copy behaviors. To do that, open the settings panel by tapping the ellipsis button and toggle "Duplicate with Behaviors" on.

Undo

To undo the last change, tap the undo button in the upper-left.

Behaviors

Any number of behaviors can be added to each object. Each behavior has a trigger and a sequence of actions. The trigger and each action can be configured.

To add a behavior to the selected object:

Behaviors are given default names like "Behavior" and "Behavior 1". To rename a behavior, long-press it in the left nav, tap "Rename", and edit the name. Since these names will become property names in code, use a camelCase name that starts with a lowercase letter.

Triggers

Each trigger has a type and a set of "Affected Objects". The supported trigger types include:

Actions

Each action has a type and a set of "Affected Objects". The supported actions include:

To change a trigger, long-press its name, select "Replace", and select a new trigger from the popup that appears.

To add another action to the action sequence of a behavior:

To cause an action to execute simultaneously with the action before it, drag it on top of the previous action and drop it.

To play a behavior in order to see its effect:

TODO: Is there a way to execute two actions at the same time TODO: rather than in sequence?

Behaviors that have missing or invalid data in their trigger or actions will have a warning icon after their name. Select the behavior in the left nav. to repair it. The trigger or actions that are invalid will have the same warning icon in the upper-right of their boxes.

To copy a behavior:

To temporarily disable a behavior:

To delete a behavior:

Images

Images can be added to a model. One way to do this is to:

  1. Copy a photo from the Photos app to the Files app using the share sheet button.
  2. Add a Frame object from the Arts category to a Reality Composer project.
  3. Edit the Frame object and select the image from the Files app.

Another way is to drag an image into a Reality Composer scene. This is easy to do in macOS, but it's unclear how to do this in iOS.

Sounds

Reality Composer provides a large selection of sounds and new sounds can be imported from apps like "Voice Memo".

To play a sound when an object is tapped:

  1. Open the Voice Memo app.
  2. Record a new voice memo.
  3. Tap the share sheet button.
  4. Save the voice memo in the Files app.
  5. In Reality Composer, add a "Tap & Play Sound" behavior to an object.
  6. In the Trigger, select an object that will be tapped to play the sound.
  7. In the "Play Sound" action, tap "Choose" for "Audio Clip".
  8. In the dialog that appears, either select a provided sound or tap "Import".
  9. If "Import" was tapped, locate a sound file in the Files app, open it, and select the newly imported sound.

Playing a Project

To play a project tap the play button in the upper-right. This will begin executing behaviors that are configured to begin when the scene starts. It will also display a stop button at the bottom center. Tap the stop button to return to editing the project.

AR Mode

In iOS (iPhone or iPad) display a model and execute its behaviors by toggling the "AR" button in the upper-right on. This will ask for permission to access the camera and them display the model on top of what the camera sees. Tap the play button to execute the behaviors.

Occlusion

To cause objects to occlude real-world objects when viewing them, tap the "..." button in the upper-left and toggle "Real-World Occlusion" to be on.

Accessing a Project

To add a Reality Composer project that created on a iPad and saved in "iCloud Drive" to an Xcode project on a Mac:

Exporting a Project

A project can be exported as a USDZ model (not easily edited) or a Reality file (easily edited).

To export the current model:

  1. Tap the ellipsis button to open the settings panel.
  2. Tap Export.
  3. Choose between exporting the entire project or only the current scene.
  4. Tap "Export" in the upper-right.
  5. Choose a location where it will be send such as AirDrop, Messages, Email, or "Save to Files".

TODO: It seems that Reality Composer now only supports TODO: exporting as a .reality file?