A 6-minute story-driven experience for the Magic Leap to show that AR is a unique and viable medium for storytelling.

The Redemption Department was created by ARTales, a 4-person team and an undergraduate scriptwriter, all looking to explore the potential of Storytelling in Augmented Reality. It started off as an experimental project and we used the building and testing of our final experience to understand how to create a narrative experience that is unique to AR.



Development Duration : 15 weeks


Platform : Magic Leap


My Role : Programmer, Animation Artist


Software Used : Maya, Unity


The Experience : The Redemption Department is a dark comedy, that takes place in the afterlife. You, as a guest are an employee in “the redemption department”, whose job is to judge the fate of spirits coming to plead their case. At the end of your work day, the tables turn, and your fate is about to be decided.

Being and interactive story, you must interact with the world and stamp the plaintiff’s file to make your choice and your choice must be made based on the information presented to you.

All the characters are morally grey and bring about conflict and uncertainty regarding the decision, thus making the story somewhat unique for every guest.



INTERACTION DESIGN :


Our team was a small one, so we were all part of the discussion for most aspects of the project. Through this I learned a lot about designing interactions for a narrative driven experience. Our goal was to focus on the narration yet keep it interactive. We were treading a fine line between story and game. It had to be handled carefully.


Controller-Stamp :

One of the first choices we made were to eliminate the use of the buttons and triggers on the Magic Leap Controller. Instead we used the controller like a stamp, with a model overlaid on the physical controller.





Since the controller was far more reliable than hand interactions in the Magic Leap, we decided it to use that as our core interaction in the story. This interaction was received well by the guests. With positive audio feedback the guests found the interaction quite satisfying and it also added to the gravitas of the decision-making process, given they were holding a physical object.



Hand Interactions :

The Magic Leap does not allow for grabbing and holding mechanisms, at least not seamlessly enough. However, it is good at registering hand gestures and key points on your palm. One of the most magical things about Headset AR, which also makes it unique your ability to see your very own hand interact with the virtual and physical world simultaneously. We tried to utilize this by implementing the use of a simple touch interaction during the story.

The experience begins with you scanning your hand on an employee recognition device and every guest was pleasantly surprised to see it work.





Each plaintiff has a file that contains their information and the Guest must touch the file to open it. This also added to the immersive nature of the story.





One of the biggest challenges we faced was that, since headset AR is still in its infancy and Magic Leap is a very recent platform, people aren’t aware of the interaction possibilities within the medium. Although we thought the absence of buttons and triggers makes the experience more intuitive, it backfired as people have now become used to pressing buttons and triggers in all virtual experiences. Therefore, we had to teach them the mechanisms before they begin the main story. Since it is a narrative experience and not a game, our clients did not want a traditional tutorial, so we had to integrate it into the story as a sort of quality assurance test before you start your day at work.





Interaction with the space :

Since AR has the ability to detect flat surfaces well, we wanted to use that in our story; the most important ones being, the walls and a table.

The table is used as the integral work area. We decided to keep its role in the experience as a table itself to maintain the integrity of the story. It acts as your employee work desk, where you can sit and listen to the plaintiffs.




The table forms the strongest and most seamless connection between the real and virtual world and emphasized the guests’ role as an employee in an office environment. Interacting with the table was very intuitive for all the guests and served the story well.

We used the physical walls to place 3-Dimensional portals that create a depth effect on the surface. The guest doesn’t necessarily interact with them, but they contain signature sounds and spatial audio, creating for a magical effect if the guest chooses to explore them. The 2 sparkly portals act as the “Good” and “Bad” place where the plaintiffs are sent once the guest makes their decision. This makes it are a crucial part of the story.





In a new medium like Augmented Reality, it is very difficult to get the guests to interact with the room, either because they are afraid to make a mistake, or they are looking for direction. At first, we wanted the guest to explore the portals and look inside them. However very few guests showed that inclination as they did not have any direct interactions associated with them. This posed a problem for us as the logic of a dialogue depended on the guests’ curiosity for the portals. In the end we had to change this and realized that in a medium like this, the guests need good reason to approach something, and if they don’t, then the experience shouldn’t depend on it.

Another challenge we faced was the limited FOV in the Magic Leap. This caused an issue when we wanted the guest to look in a particular direction and they wouldn’t. Spatial audio helped us only with left and right direction, but not up and down. This forced us to adjust the size and location of certain objects, as well as the pacing of different events in the story to attract the guest’s attention.

Designing for an interactive story was quite a challenge and taught me how to deal with a new medium, limited field of view and how to best harness the advantages of Augmented Reality. We may not have implemented all of them in our experience, but we learnt a lot about the dos and don’ts of the process.

I am hoping that in the future, Augmented reality with support the feature of Hand Occlusion which will enhance the guest’s immersion and change the nature of interacting in AR.

Like this, as technology progresses, our approach toward AR and its interaction design will also have to adapt.


PROGRAMMING :


When we started this project, we spent the first 4 weeks researching the technical limitations and advantage of various AR platforms. I was put in charge of the Magic Leap and became quite familiar with the APIs. I spent a lot of time exploring the different possible interactions with both hands and controller.

I experimented with the key points on the hand that the MLHands API is able to detect




I was able to attach an unmeshed collider on the hand that could be used to detect when the guest’s palm touched another virtual object. I also tried to use the key point to attach virtual objects to the hand allowing for grabbing. Unfortunately, the object would get displaced if the hand goes out of the headset field of view, therefore, this made the grabbing very unreliable. Hence, we stuck to invisible colliders, and only used touch mechanisms.




Once our story draft was in place and we had chosen our platform, I had to make quick prototypes to test the possibilities of the elements in the story. The stamp was a crucial part of that.

I was very determined for it to be realistic and satisfying. It should feel like an actual stamp. I spent a few days trying different techniques. The superimposition with the controller was easy enough, but the movement of the shaft needed to respond well to the guest’s movement.




At first, I tried creating an animation, but the pacing wasn’t right. Then I attached a rigid body collider to the lower shaft and connected it to a fixed joint so that it would move only along the controller axis when it collides with a surface.



To stamp the file, the stamp handle collider would change the paper texture upon collision.




Experimenting with collider and rigid body physics was a lot of fun and seeing the stamp work flawlessly was the cherry on top.

Another feature I spent time on was using particle systems to direct the attention of the guest. The Magic Leap headset acts as the camera and I made a script that repeatedly moves a particle effect from the guest’s front vision to the object in question. We playtested this with a number of guests, trying to direct their attention to the portals and it worked quite successfully after many iterations. Eventually, we did not use this system, but given a medium like AR with such a limited FOV, I felt it was a good use of my time, trying to get it to work.

Programming for a platform like Magic Leap was quite exciting. It was very interesting to experiment with different interactions and understand the challenges of working with an AR platform. I feel confident in my familiarity with headset AR and look forward to developing for such projects in the future.


ANIMATION :


My 2nd role on this team was an animator. I had never worked on an animation project this large before and it certainly came with its challenges.

The characters had to be quite expressive but since in a medium like AR, the space is dynamic, all the animations had to be in place. I couldn’t move the character around too much, in case they bumped into a table or wall. There were 4 characters in total and I had to rig and animate them.


Juicy : She is a flirtatious character who has flair for the dramatic. I tried to focus on her hand movements and emphasize her sass.



Mochi : He is lethargic and has an interesting tone of voice. I tried to match his body language to his words and draw attention to the fact that he is tried and fed up.



Lumps : He is a Momma’s boy and a cry baby. All I had to do was make him throw a tantrum.



Manager : She was the first character I rigged and animated for the project. She is a bit stiff but I decided to keep her that way, as it suited her stoic and stern character. Her round body posed a challenge to create a variety of movements, but I tried to match her voice inflections with her body rotation to emphasize her words.



Having experienced, voice actors from the school of drama certainly helped with the emotion of the characters and added to their performance.

The rigging of the characters was a quite problematic, especially since rigging is not my forte. I tried to keep them simple, but still tried to fulfill the animation requirement of each character. We faced some issues with the blend weights in the Magic Leap and I spent quite a bit of time trying to fix it. This certainly took away from my time to animate but It taught me how to animate quickly. I definitely wish I had more time to polish the animations, but It helped me understand how to portray the nature of the characters through body movements and most importantly, the challenges of animating in a medium like AR.


MY JOURNEY WITH NARRATIVE AR :


I may have officially been a programmer and an animator for this project and I did spend most of my time doing those things. However, the most valuable thing I took away from this project is the understanding of Augmented Reality and its place in the world of new media.

More than creating an experience that rivals other platforms, if we focus on its features, AR has a better chance. Moreover, if we capitalize on its connection to the real world and use that to our advantage, AR can be a formidable medium.

For eg: Something that we couldn’t achieve during our project was making the interaction between the real and virtual world absolutely crucial to the story. Yes, opening the folders with your hand is very immersive and unique to AR but it wasn’t integral to the story.

In addition, not every kind of story works with Augmented Reality, and the script must be specific to AR. The fact that every room is unique and can change the story is something that needs to be taken into consideration and taken advantage of when dealing with this medium, not just seen as a problem.

Overall, this was a big learning experience for me and I can say with absolute certainty that I now understand Augmented Reality a lot better than I did before. I look forward to working more with it.