Creating Augmented Reality iPhone app experiences – getting used to Apple’s ARKit

Recently as part of our internal innovation program which we call 'WaveLabs', the team explored Apple's Augment Reality Framework, called 'ARKit'. We reviewed the technology as well as tackling the challenges of designing Augmented Reality experiences for web and mobile apps. 

Augmented Reality is a combination of real-world and computer-generated elements that can be used to bring a real-time interactive app experience to users. A view of the real world is enhanced or augmented with digital elements. At Wave, we can incorporate Augmented Reality into your app.

ARKit is a software toolkit released by Apple for iOS devices. This allows developers to design Augmented Reality experiences in iPhone apps, without the use of additional hardware.

ARKit Features

World Tracking

This is the key feature that Augmented Reality provides. Using camera images and inputs from the phones sensors ARKit will track the environment you are in. This allows an ARKit enabled app to accurately track the users position in a room, including the direction that you are facing, seen here with the yellow dots.

Scene Understanding

As well as World Tracking ARKit also provides functionality to detect physical features in the world it sees. This is key to providing more advanced Augmented Reality experiences where the user can interact with the physical world via the app.

One of the key Scene Understanding features provided by ARKit is “horizontal plane detection”. This is a fancy way of saying ARKit can detect flat horizontal surfaces, such as a table-top or the floor. This allows us to place virtual objects in the world that appear to sit on real surfaces that appear in the world.

Rendering

The final part of ARKit technology is support for rendering Augmented Reality scenes. This is provided by way of integrating with Apple’s existing 3D engine known as SceneKit. This allows developers to leverage existing technology & content to deliver Augmented Reality experiences.

Review of ARKit

In our experience, we found the World Tracking to be very accurate. We ran several experiments in our office and found it could track our environment without problem.

Our experiments consisted of placing markers in the Augmented Reality world, rotating, moving several meters and returning to the staring position. We observed that the markers were still in (or very close to) their original positions, indicating that ARKit was able to track our environment accurately.

We found the horizontal plane detection to be good, although it seems to only detect when you are within 1m of the target. The detected distance are ok too – checkout AR Ruler 3D – an app based on ARKit that has some pretty cool measurement features.

There are of course limitations to ARKit. If the physical environment has low detail, for example a room of plain-white walls, ARKit won’t have any detail to lock on to and will result in degraded performance. Similarly, if the user violently moves or shakes the phone ARKit will get mixed up. We found best results when the user treated the device like a camera view finder to look around the Augmented Reality world.

All of this AR technology demands a fair amount of processing power and I expect it would drain your battery pretty quick.

From a developer point of view, we found the ARKit API’s to be easy to use and understand. It helps to have background in computer graphics as well as other iOS app frameworks.

Aligning to the real world

A key point to understand about Augmented Reality and ARKit is that tracking is relative; positions are measured in meters from the point where the user started the Augmented Reality app. Out-of-box ARKit does not align these positions to a common frame of reference e.g. GPS coordinates.

There are many Augmented Reality applications that would benefit from this type of alignment. This allows pre-populated content to be loaded in the correct position without user input. This can be achieved by combining ARKit with location tracking technologies.

Examples of location based technologies that could be integrated with ARKit to produce an expanded Augmented Reality experience:

• GPS tracking – This can give you location to +/-10m in clear outdoor environment.
• Phone’s built in compass – ARKit does have the ability to use this, however it’s accuracy is poor indoors.
• Bluetooth beacons – also suitable for indoor environments. Beacons can operate in a ‘Ranging Mode’ to give the users location to 1-4m.
• Aligning to a fixed point for reference – eg QR code. This is popular in other Augmented Reality solutions and is suited to indoor environments. This is the most accurate method however this requires the environment to be pre-calibrated and requires action by the user to perform the calibration.

The main issue is that most location tracking is far less accurate and consistent than ARKit’s World Tracking. This means that although we can roughly align to the real-world, it’s difficult to perfectly align pre-populated content without the user preforming a calibration or adjustment step.

One way this could be overcome is to gamify the calibration step so the user does not feel launching the Augmented Reality app is a chore. Otherwise with creative design and production small inaccuracy’s might not detract from the app experience. A few meters discrepancy in GPS coordinates might not matter if you are placing a label above a landmark several km away.

Seen here in our Wave Labs demo app, we are plotting landmarks that are visible from the office balcony. The app uses the compass to get fairly accurate alignment, however we did need to manually adjust the app by 1-2 degrees to have the labels accurately aligned to the buildings. In this case we aligned the crane, once this initial calibration is done all other labels a plotted correctly without the need to adjust.

Designing Augmented Reality iPhone app experiences

There are many considerations to take into account when designing Augmented Reality experiences. We have found ARKit’s World Tracking to be accurate and consistent. This is great for designing games and interactive experiences like IKEA Place that allows the user to interactively place virtual furniture in a room.

There are also exciting opportunities to combine Augmented Reality with location based technologies. This is suited to wider scale experiences such as virtual tours or identification of landmarks. There are some challenges in achieving this type of integration however if the limitations are taken into account they can be worked around creatively.

Interested in Augmented Reality? Talk to us.