ResearchInteraction DesignAIIEEE

CuriosityXR

Point at anything in your room and learn about it in mixed reality. #1 AI learning app on Meta Quest, 1,000+ copies sold.

1,000+ copies sold
#1 AI learning app on Meta Quest
IEEE VR 2023 paper
AIxVR 2024 paper

Contextualize learning with Mixed Reality

For my Master's thesis, I designed and built a platform to support curiosity with XR. Learners would be able to learn math concepts such as the volume of a cylinder when looking at a cup or how plant cells look like when near a plant.

Tools & Software

Meta Quest Pro & HoloLens 2

Unity, MRTK, Oculus Interaction SDK

Socket.io, PyTorch

XR Researcher @ Adaptive Context Environment Lab

Supervised by Dr. Alexis Morris, Dr. Ian Clarke

MDes. Thesis Research

How can XR & AI enable curiosity-based learning?

This research aimed to explore how Context-aware agents could support curiosity and knowledge recall through Mixed Reality.

Augmented reality overlays on a landscape, from a TEDx talk on AR in education — How Augmented Reality Will Change Education Completely | Florian Radke | TEDxGateway

Process: three research sub-questions

Explored the research question through different prototypes and sub-questions focusing on these aspects.

Acquiring context from XR devices

We explored using different headsets and devices to acquire user context, visual context, speech context useful for creating and designing educational experiences.

Creating curiosity-support agents for learning

There are many possible ways to create support agents to help users learn. We used AI, LLM and other APIs to explore various prototypes.

Evaluating the effectiveness of XR in supporting learning

A user study was conducted at the end of the design research project to understand how XR compares with other traditional learning interfaces.

Methodology

A loop for each prototype from ideation to evaluation.

Project timeline across the prototypes — Timeline

Literature Review

We started our process by exploring the current papers and prototypes in the research field.

Map of the papers and prototypes reviewed

Gaps for opportunity

1. Potential to implement context-aware systems using different learning theories for new MR interfaces for educational purposes.

2. Potential to explore Artificial Intelligence & educator tools to support the creation of learning content for MR headsets.

Prototype 1

Goal: help learners visualize

This idea was inspired by a TV show, Shaka Laka Boom Boom where the protagonist uses a magic pencil to sketch and then it becomes real.

A sketched apple becoming a 3D object in mixed reality

Process: building the prototype

Tools: HoloLens 2, paper and pencil. Unity (MRTK3), Python (PyTorch), HoloLens Remoting.

To keep the interaction intuitive and ubiquitous, the idea was to use hand tracking and have no buttons or interface for the user, to build an interaction which automatically detects the user's intention, the time drawing is completed and automatically bring the object in front of the user.

Prototype 1 system architecture — Architecture

The real effort was put in the design of how the interaction should work, we had several discussions and ideas to execute the prototype for the interaction where the user would simply draw on a piece of paper. The entire architecture was built to detect the drawing at the right time and go through API's to get the object on top of the sketch for the user.

Outcome: a viral LinkedIn post :)

It was indeed a magical experience, the experiment allowed to get camera stream for visual context such as object detection, scene understanding and present hand-interactive 3D models in Mixed Reality.

LinkedIn post

Prototype 2

Goal: visualization with speech and hand interactions

This research was inspired by a superpower which could help visual learners with better context.

Process: early sketches to visualize the concept

Sketch of asking an LLM about an object in view

Sketch of pointing at an object and asking about it

Process: building the prototype

Tools: HoloLens 2. Unity, MRTK3, SketchFab API, Wit.ai.

The device allowed access to camera stream and continuous speech recognition which was important to prototype this interaction experiment.

Prototype 2 system architecture — Architecture

First layer was to identify the user's intent, before GPT and LLMs, Wit.ai was a great tool for this. Then based on the multimodal input of the user, the system would identify a 3D model best relevant for the learner and use SketchFab's API to fetch it and present it to the learner.

Outcome: play with XR toys while learning about them

I tested the prototype with other researchers in the lab and advisors, it was a great experience to create an XR space with the speed of speech.

3D models summoned by speech in mixed reality

Prototype 3

Goal: show contextual learning content

This prototype was aimed to test if interactions with the real-world could be important for better memory retention.

Process: early sketches to visualize the concept

Sketch of a basketball's trajectory shown in XR

Physics, Math

Learners could play with a basketball and see the trajectory in XR, analyze, understand and ask questions about it.

Moving objects, real-time

Sketch of learning content anchored to a plant and a table

Chemistry, Math, Biology

Science is deeply related to the objects in the environment and there is so much we can learn in the context from common objects like plants (growth rate), tables (Free Body Diagrams).

Still objects, exploration

Process: building the prototype

Tools: Meta Quest Pro, Zed Mini. Unity, Oculus Interaction SDK, GPT-3.

Quest Pro released in 2023 and had a much better immersive experience than that of the HoloLens 2, but unfortunately it does not provide camera access so we had to place an external Zed Mini sensor for depth and 3D understanding of the user's environment.

Prototype 3 system architecture — Architecture

The agent would automatically create a Mixed Reality lesson and present it to the user based on the objects in the environment using LLMs and object detection in 3D to contextually anchor the learning content.

Outcome: learn something new when you are near objects

A lesson anchored to a real object in mixed reality

Final Prototype

Goal: build a multimodal learning experience

This research aimed to explore how Context-aware agents could support curiosity and knowledge recall through Mixed Reality.

Process: designing an educational contextual interface

Understanding the user journey

This research aimed to explore how Context-aware agents could support curiosity and knowledge recall through Mixed Reality.

User journey through the CuriosityXR interface

XR templates for contextual UI

Sketches to visualize various anchored UI for XR.

Sketches of anchored UI templates for XR

Process: building the prototype

Tools: Meta Quest Pro.

To keep the interaction intuitive and ubiquitous, the idea was to use hand tracking and have no buttons or interface for the user.

Final prototype system architecture — Architecture

User study & research: 2 hrs of in-person study

REB-approved research study sessions were conducted to understand the effectiveness of using Mixed Reality to learn about a plant cell compared to using a browser on a laptop.

A participant in the in-person user study

Visual retention improvement

Ratings for CuriosityXR by 6 user participants. Engagement level, effectiveness in aiding learning and ease of use were measured.

Participant ratings for engagement, effectiveness and ease of use

Outcome: 100+ people loved CuriosityXR

People from all ages visited OCAD during a 3-day exhibition where they were able to use CuriosityXR and learn about different objects through a Voice interface.

Thesis Exhibition @ OCAD University