CuriosityXR
#1 AI Learning app on Meta Quest. 1,000+ copies sold.
Contextualize learning with Mixed Reality
For my Master's thesis, I designed and built a platform to support curiosity with XR. Learners would be able to learn math concepts such as the volume of a cylinder when looking at a cup or how plant cells look like when near a plant.
Tools & Software
Meta Quest Pro & Hololens 2 Unity, MRTK, Oculus Interaction SDK Socket.io, PyTorch
XR Researcher @ Adaptive Context Environment Lab
Supervised by Dr. Alexis Morris, Dr. Ian Clarke
Acquiring Context from XR devices
We explored using different headsets and devices to acquire user context, visual context, speech context useful for creating and designing educational experiences.
Creating curiosity-support agents for learning
There are many possible ways to create support agents to help users learn. We used AI, LLM and other APIs to explore various prototypes.
Evaluating the effectiveness of XR in supporting learning
A user study was conducted at the end of the design research project to understand how XR compares with other traditional learning interfaces.
A loop for each prototype from ideation to evaluation
1. Potential to implement context-aware systems using different learning theories for new MR interfaces for educational purposes.
2. Potential to explore Artificial Intelligence & educator tools to support the creation of learning content for MR headsets.
Help learners visualize
This idea was inspired by a TV show, Shaka Laka Boom Boom where the protagonist uses a magic pencil to sketch and then it becomes real.
To keep the interaction intuitive and ubiquitous, the idea was to use hand tracking and have no buttons or interface for the user, to build an interaction which automatically detects the user’s intention, the time drawing is completed and automatically bring the object in front of the user.
The real effort was put in the design of how the interaction should work, we had several discussions and ideas to execute the prototype for the interaction where the user would simply draw on a piece of paper. The entire architecture was built to detect the drawing at the right time and go through API’s to get the object on top of the sketch for the user.
Viral Linkedin Post :)
Linkedin PostIt was indeed a magical experience, the experiment allowed to get camera stream for visual context such as object detection, scene understanding and present hand-interactive 3D models in Mixed Reality.
Visualization with speech and hand interactions
This research was inspired by a superpower which could help visual learners with better context.
The device allowed access to camera stream and continuous speech recognition which was important to prototype this interaction experiment.
First layer was to identify the user's intent, before GPT and LLMs, Wit.ai was a great tool for this. Then based on the multimodal input of the user, the system would identify a 3D model best relevant for the learner and use SketchFab's API to fetch it and present it to the learner.
Play with XR toys while learning about them
I tested the prototype with other researchers in the lab and advisors, it was a great experience to create an XR space with the speed of speech.
Show contextual learning content
This prototype was aimed to test if nteractions with the real-world could be important for better memory retention.
Physics, Math
Learners could play with a basketball and see the trajectory in XR, analyze, understand and ask questions about it.
Moving objects, real-time
Chemistry, Math, Biology
Science is deeply related to the objects in the environment and there is so much we can learn in the context from common objects like plants (growth rate), tables (Free Body Diagrams).
still objects, exploration
Quest Pro released in 2023 and had a much better immersive experience than that of the Hololens 2, but unfortunately it doen not provide camera access so we had to place an external Zed Mini sensor for depth and 3D understanding of the user's environment.
The agent would automatically create a Mixed Reality lesson and present it to the user based on the objects in the environment using LLMs and object detection in 3D to contextually anchor the learning content.
Learn something new when you are near objects
It was indeed a magical experience, the experiment allowed to get camera stream for visual context such as object detection, scene understanding and present hand-interactive 3D models in . Mixe d Reality.
Build a multimodal learning experience
This research aimed to explore how Context-aware agents could support curiosity and knowledge recall through Mixed Reality.
Understanding the user journey
This research aimed to explore how Context-aware agents could support curiosity and knowledge recall through Mixed Reality.
XR Templates for Contextual UI
Sketches to visualize various anchored UI for XR
To keep the interaction intuitive and ubiquitous, the idea was to use hand tracking and have no buttons or interface for the user.
2 hrs of in-person study
REB-approved research study sessions were conducted to understand the effectiveness of using Mixed Reality to learn about a plant cell compared to using a browser on a laptop.
Visual retention improvement
Ratings for CuriosityXR by 6 user participants. Engagement level, effectiveness in aiding learning and ease of use were measured.
100+ people loved CuriosityXR
Thesis Exhibition @ OCAD UniversityPeople from all ages visited OCAD during a 3-day exhibition where they were able to use CurioistyXR andd learn about different objects through a Voice interface.
Publications
IEEEVR 2023
Research paperPublished the initial interaction design concept.
AIxVR 2024
Research PaperFull-paper publication with technical and all prototype details.
Launch on Meta Quest Store
Support and Accelerators
