The project was inspired by the ancient mnemonic technique of the "Memory Palace," as detailed in Frances A. Yates’ book, The Art of Memory. This method involves mentally mapping information to physical locations to aid in memory recall. The team, driven by a shared enthusiasm for XR Design, saw an opportunity to modernize this technique using Mixed Reality, with a focus on language learning. They chose to create a prototype for learning Spanish, though the method is adaptable to other subjects.
The initial design was made in Figma. 3D objects were created in Blender and then transformed and developed in Unity. The visual assets were made with Adobe Illustrator, Adobe Photoshop, Adobe After Effect and DOTween (animations).
After brainstorming, we settled on 3 key features we thought would make the exercise of learning a language more interesting and immersive:
Create: This is the default mode of the experience. The user is able to create a flashcard, using their voice to input the word (in English) they would like to have on the card. They can then place that card anywhere in the environment.
Randomize: Gives the user a pre-configured flashcard to place in the environment. These flashcards include native speaker audio playback and the option to load a 3D model representation of said object if the user wishes (or doesn't have said physical item in their room). The 3D model & flashcard are linked and follow each other if either is moved.
Practice: This quiz feature is intended for after the user has placed a few flashcards. All flashcard text in the environment becomes temporarily hidden, and the user is prompted with a Spanish word and points at the flashcard object they believe the word represents.
All of the flashcards and associated 3D models are persistent between sessions. The three modes listed above are launched via the wrist-button activated menu panel. The app is hand tracking only and requires microphone permission to use
the flashcard creation feature.
Spatial Anchors: The foundation of Loci. A powerful and reliable feature for mixed reality experiences, opening up a host of possibilities for app creation using Quest Passthrough.
Wit.AI via Meta Voice SDK (Dictation): We leveraged Meta's Voice SDK's Dictation feature to allow the voice-search during flashcard creation. This is activated by a flashcard button press, and returns the first word recognized by the audio input, comparing it against the MLCommons dataset which returns the Spanish translation. One challenge was that there are many duplicate words in the dataset, so voice dictation alone isn't sufficient to get perfect translation accuracy. Extending this feature to use Voice SDK's more advanced AI-powered voice interpretation could take it to the next level.
Visual Design Tools: Blender, Figma, DOTween (animations) among others were used to craft the visual language of Loci.
For development, one challenge involved allowing the user to move the flashcards while still supporting spatial anchors. Once a spatial anchor is created it can't be moved. Our developer was able to create a script that deletes and re-adds the card's anchor and all references to it whenever an object is grabbed and re-positioned, respectively. We found that for our use, it was performant enough and did not cause any frame drops.
Another challenge involved creating a wrist-based UI while dealing with hand occlusion. The wrist menu button sits like a watch face on the hand tracking wrist bone. We found that once the menu panel was expanded, its position needed to be un-parented to the tracked hand to avoid jittering whilst the opposite hand interacted with the panel. We settled on a solution to have the panel free floating once activated, gently following the users gaze.
Inspired by the "bubble" motif of the art style, we developed an unobtrusive user onboarding / hint system we believe is perfect for mixed reality. A semi-transparent hint "bubble" very gently follows the users head gaze. It generally stays in the users field-of-view (assuming no rapid change in head position/rotation) but is designed not to linger in front of the user's face, rather off to the side. We also let the user "pop" the bubble if they want to get rid of it. as we want the user to maintain control over their environment.
As a team, we had to learn how to work together spanning across time zones (from California to New York). For some of our designers, it was an opportunity to explore further with tools such as Unity and Blender. For our developer, this was a chance to sharpen skills related to developing in mixed reality using Presence Platform, and unlocked ideas for future apps.
Thank you for your time! :)
