Our initial concept envisioned a showcase of television models, each representing a distinct era—past, present, and future. Users would interact with floating UI panels to navigate the system and explore the history and interaction methods of each period. However, this idea came with many problems.
Design & Develop
An Interactive Learning System for Exploring Overlooked and Unfamiliar Forms of Interaction with Visual Media
About
Since the 1950s, digital visual media has evolved from the first digital scanner to innovations like television cameras and coloured images. While fewer people use traditional TVs today, the technology continues to advance, including gesture-controlled TVs—though many are unaware of them. This project introduces users to both older and emerging TV technologies through XR (Extended reality) experiences, enabling learning through interactions rather than static text or images.
This project's core goal is to address the lack of contextual understanding surrounding television history - identified from our visit to ACMI [1] - by creating an interactive learning system that facilitates learning through XR technology.
Design Process
Our project followed an iterative design thinking process, the double-diamond method [2]. We began with secondary research to explore possible ways to extend traditional learning methods. We had multiple brainstorming sessions and conducted a literature review to understand television history and XR technology, which guided the direction of our research. Additionally, we conducted a fieldwork study to gain key insights and formed a clearer understanding of participants television consumption, adoption of XR as a learning tool and concluded our first milestone.
Our second milestone started with the design and development phase, where we used different techniques to created different high-fidelity prototypes on ShapesXR and Unity. These prototypes were created con-concurrently and had gone through multiple iterations to achieve a immersive experience. The final milestone focused on user testing when we validated the prototypes design and evaluating the user experience.
Problem Statement
How might we support people in discovering and interacting with overlooked or unfamiliar forms of visual media?
How might we make learning about interface evolution immersive, fun, and accessible even for XR beginners?
Iteration 3 - Hardware Integration
Iteration 4 - Unity Prototype
7
interview participants
10+
questions were asked
Interviews were conducted in a face-to-face setting with a guide to support the interviewer. These interviews provided genuine user insights, capturing their emotions, desires and challenges. Meeting with 7 individuals (with technical and non-technical backgrounds) played a crucial role in shaping our system’s vision.
XR will be helpful for all different disciplines, but it will benefit more from areas that require you to visualise
I think TV remote controls do a good job for their usage, where usually you don't need to input a lot of text
Using XR for education is actually great, but it's expensive, especially for individuals, but looking at the benefits, I think that will be great, because it gives you better immersive experience.
I think XR is easy for users to learn, and you can add more materials inside, and it will make the learning materials more interactive
I prefer voice control and gestures because it offers more autonomy. It automates the process, and it's more of into the device that I just give a voice command and it processes my command and gets stuff done with it
Semi-structured Interviews
Surveys
25
people took the survey
23
questions were asked
Consuming visual media content benefits from intuitive and familiar input methods, particularly for accessibility.
Portable screens are the preferred medium for visual media consumption
XR enhances knowledge visualisation, making learning more accessible
XR technology is underutilised as a learning tool
Gesture control in XR can be integrated with other input methods for a more seamless experience
Physical controls remain more popular than non-physical alternatives.
Survey Insights
Brainstorm
Based on the data from secondary research and fieldwork, we drew 3 sketches to inform our design requirements. Ultimately, we went with sketch 3 due to its tangible and immersive nature
Personas
Storyboard
Low-fidelity Sketches
Mid-fi Prototyping
Based on the data from secondary research and fieldwork, we created two personas, one represent people with technical background and the other is for non-technical individuals. Personas allow for a more-in-depth understanding of the user needs, goals, behaviour and pain points.
Storyboard was used to visually plan how the users would use our system. The storyboard highlights each step the user would take to explore the transition of interactions, from tangible to gesture-control. This approach help us set aside our internal biases and stayed focus on understanding and empathising with user behaviour, keeping us mindful of what drives their actions.
Sketches were created to explore how the system work and visualise the user journey
After we settled on the idea, we did rapid prototyping with ShapesXR to design the UI and Spline for animation to quickly bring our idea to life.
ShapesXR
Spline
Interation 1 - UI & Viewing Direction
Through team discussions, we identified that the horizontal UI panels were too cluttered with text and occupied excessive navigation space, which could negatively impact user engagement. These problems could potentially hinder user engagement. Therefore, we decided to switch to a vertical layout and condense the text on each panel for a clearer and more engaging experience
Initially, we planned to include a third television model, a screenless television that represent the future by connecting a bluetooth keyboard to the VR headset. However, due to time constraint, we were unable to implement this successfully.
One of the major challenges we faced in developing our Unity prototype was working with the hardware components. Throughout the process, the dials on the miniature TV model frequently detached, and several components failed to function as intended, requiring resoldering and hardware adjustments. Even after successfully connecting the miniature model to the Unity software, we encountered noticeable input delays between the physical interactions and their corresponding outputs in Unity.
We initially believed ShapesXR could support more complex interactions, but discovered it is limited to basic digital interactions and does not support MP4 file playback. Since our project involves hardware integration and gesture control, we turned to Unity to implement gesture control, video playback and enable hardware connectivity. Additionally, we later found that ShapesXR does not allow screen sharing during usability evaluations, which presented missed opportunities to collect valuable feedback.
As a result, we used ShapesXR primarily as a design tool to visualise the UI, while Unity was employed to demonstrate and test the user experience.
To evaluate our system, we utilised surveys and usability tests to validate the prototype design and assess the user experience.
We collected a total of 13 survey responses from visitors at the Endeavour Exhibition, following their interaction with our project. These surveys gathered feedback on their experience using the prototype and their overall learning experience through Mixed Reality.
Usability testing was conducted with 12 participants, using moderated observation combined with the Think Aloud method to capture participants’ thoughts and decision-making processes. We observed real-time interactions and collected feedback on usability, functionality, and general user experience. After completing the tasks, each participant answered a set of post-task questions focused on three key areas: clarity of instructions, system engagement, and confidence in navigating the system independently.
Before
Before
Before
After
After
After
Project Setup
System UI
Iteration 2 - Down to 2 television models
High-fi Prototyping
Actionable tasks
Navigate the learning UI
Give feedback on system UI and navigation flow
Observables
Do users navigate the tasks correctly?
Can they see UIs and the miniature TVs?
Do they request further instructions?
Design Evaluation
Intuitiveness of control: Do the knobs naturally encourage users to interact with them?
Playful discovery: Do the tasks and prototype invite curiosity and exploration without guidance?
User confidence: Do participants feel capable and comfortable navigating the experience independently?
Actionable tasks
Navigate the learning mode on Unity
Turn on/off the flat TV
Gesture controlling the TV (Swipe, push, point)
Observables
Can participants start without verbal help?
Does the system recognise the participant’s hand gestures?
Participants’ struggles while getting familiar with gesture-control interactions
Design Evaluation
Feedback during gestures: Do the participants receive instant feedback while going through the task
Task Design - Retro TV
Task Design - Gesture-control TV
Secondary Research
Primary Research
7 Interviews + 24 Surveys
To better understand the gap between user needs and past research, we conducted interviews and survey questionnaires. Participants were asked about their visual media consumption and their general knowledge of XR technologies.
All interview participants provided informed consent via a consent form that allowed their audio recordings to be transcribed and analysed.
Younger generations increasingly using smartphones and social media instead of conventional television
Younger people are unfamiliar with older models
XR in visual media education is still limited
XR-based learning enhances motivation and learning outcomes more effective than traditional learning methods
XR technologies offer immersive learning experiences, making them powerful tools for engagement and retention
Gesture-control television are intuitive and engaging but also present learning challenges
Literature Review
Media consumption has changed
Television Literacy
XR challenges with integrated audio-visual cues
XR-based learning
XR technologies
Gesture-based television
Semi-structured interviews
7 participants
Facet-to-face
Interview guide
Surveys
24 respondents
Closed and open-ended questions
Survey questionnaires
Solution
An Interactive Learning System for Exploring Overlooked and Unfamiliar Forms of Interaction with Visual Media
During our fieldwork study, we identified that people nowadays rarely use televisions to consume visual media and show low awareness or adoption of gesture-based controls and XR technologies, especially as learning tools. Our project aims to bridge this gap, by expose users to unfamiliar technology like gesture-based control television and older technology with a showcase of several digitalised television models using Meta Quest 3 VR headset.
ShapesXR prototype for showing the onboarding instructions and the system’s UI
ShapesXR was utilised to design the user interface, offering a clear visual representation of the system’s structure and functionality. It also enabled the creation of immersive experiences by integrating 3D objects and spatial interactions, allowing users to engage with the content in a more intuitive and interactive manner.
To achieve more a realistic experience, Unity was utilised to replicate real TVs interactions. Using the same TV models from ShapesXR, this prototype aims to bridge the gap between the physical and virtual worlds. We began by acquiring miniature TV models and identifying the necessary hardware components for the project.
Through iterative development and integration of the Meta Quest 3 toolkit and its Pass Through feature, we built a mixed reality system that have both tangible and virtual interactions—allowing users to change channels and adjust volume using a physical dial on a miniature retro TV while seeing these changes reflected on a virtual digital twin.
In addition, we successfully implemented gesture-controlled functionality, enabling users to operate the TV using hand gestures.
Through guided instructions via the UI panels and virtual buttons, users can navigate and gain a better understanding of how old television models works
Moving on to a more present television model, users will get to test gesture-control technology that offers a unique experience to control TV at the wave of their hands
Unity prototype replicating the interactivity and realistic experience of using TVs
2 Prototypes
ShapesXR Prototype
Unity Prototype
Part 1 - Retro TV Interactions
Part 2 - Gesture-control TV Interactions
User Evaluation Methods
Esp 32
Breadboard
3D Printed Knobs
XRTV - Extended Reality Television
INFO90009
Discover
Define
Solutions
Design & Develop
Evaluation
After gathering data from both surveys and interviews, we analysed data and used thematic analysis to group and assign codes based on recurrent themes and patterns. Additionally, we used affinity mapping to make sense of large volume of quantitative data.
Affinity Mapping & Thematic Analysis
Finding 1
Users prioritise simplicity and comfort of old technologies over new technologies
Thematic analysis showed that participants rarely use TV for visual media consumption and most were unaware of gesture control technology. 75% of participants preferred traditional remote controls due to their ease of use and familiarity.
For me, a traditional remote control is the easiest to use and a Smart remote control makes me feel comfortable because it's similar to when I type text on a smartphone.
I don't think I have like beyond conducting my own research and sometimes entertainment, I don't think I've used it for learning
Finding 2
Low awareness of adopting XR as a learning tool
Surveys found that nearly 45.8% of respondents were familiar with XR technology, while 41.7% were aware of it but had never used it. Despite this awareness, XR remains underutilised as a learning tool, likely due to insufficient knowledge about its educational applications, limited available content, or the challenging learning curve required to master XR interactions.
Conclusion
References
Our proposed system is grounded in user research and developed through two prototypes, each individually emphasising on user interface and user experience. The evaluation results underscore the educational benefits of interactive and immersive learning, especially for young children. A major challenge in this project has been designing for XR, which requires seamlessly integrating the interface with both human interaction and real-world objects and environments.
Looking ahead, we plan to refine the system and enhance hardware integration to build a more comprehensive learning environment. This includes conducting interviews and cognitive walkthroughs with experts in education and emerging technologies to deepen our understanding and improve the project. Due to time constraints, only two television models were included in the current prototype; future versions should feature a broader historical range, incorporate audio and visual feedback, and expand the selection of TV models to better illustrate the evolution of visual media.
[1]https://www.acmi.net.au/education/school-program-and-resources/texts-stories-and-culture-australian-tv-miniatures/
[2]https://www.hi-interactive.com/blog/ux-process-the-double-diamond
Evaluation Findings
Clarity, visibility, control understanding and system feedback
Engagement and challenges with physical controls
Engagement and challenges with hand gesture controls
Limitations of XR Technology
Learning Experience & Immersion of Extended Reality (XR)
Recommendations for Future Iterations
“I want to see the following...”
It became evident that certain interface elements required refinement to enhance user experience by improving flow and minimising inconvenience.
During the evaluation, users pointed out several issues: the text size was too small, the instruction panel lacked feedback upon task completion, and the system did not provide an instructional overview before interaction—leaving users unsure of where to begin. Additionally, contrary to our initial assumption that a vertical layout would offer a better experience, some users reported neck discomfort and suggested a horizontal layout instead.
Participants generally found the interaction with the physical retro TV model intuitive and engaging, appreciating its nostalgic appeal. However, confusion arose due to the unclear separation between physical and virtual controls, with several users attempting to interact with virtual elements instead of the physical model.
The small size of the model also made it easy to miss, reducing visibility and immersion. Additionally, mismatches between the physical dial positions and virtual outputs caused minor usability issues. Despite these challenges, users remained engaged, highlighting the potential of combining tangible interaction with mixed reality. Future iterations will focus on improving control clarity, model size, and input-output alignment.
Gesture control was found to be more challenging than physical dials, with a clear divide between participants who understood the gestures and those who struggled. While users who successfully performed the gestures found the interaction engaging and enjoyable, others faced difficulties due to inconsistent gesture sensitivity, overlapping gesture recognition (e.g., swipe down vs. push), and lack of clear visual feedback.
Participants were also unsure of the correct hand placement and timing for gestures, as hand positioning guidance was not yet implemented in the Unity prototype. Despite these usability issues—mostly related to software—participants viewed gesture control as a fun and highly interactive feature with strong engagement potential.
While XR offers immersive and engaging interactions, several limitations were observed during usability testing. New users, particularly those unfamiliar with VR headsets, often felt overwhelmed and required additional guidance. Hardware-related issues such as headset weight and cybersickness remain major concerns—causing discomfort, neck strain, dizziness, and nausea, especially with prolonged use.
Additionally, the prototype setup demands a safe and spacious environment. In crowded or constrained areas, headset calibration may be disrupted, and users may struggle to perform necessary gestures or movements. These factors highlight the need for shorter experiences, careful space planning, and user support for effective XR implementation.
The findings support the project’s goal of enhancing learning through extended reality. Participants reported a highly positive overall experience with the prototype, particularly enjoying the comparison between retro and modern TVs. While users were initially more focused on interacting than reading informational panels, engagement increased significantly once they began the ‘Learn’ function. Gesture control was seen as more engaging due to its interactive and playful nature.
Most participants agreed that active, hands-on learning was more effective—especially for children—compared to traditional methods like reading or lectures. Participants also saw potential for XR in other educational contexts, such as safely exploring science experiments. The prototype was considered accessible even to users without prior XR experience, though improvements—such as shortening long text—are needed to further enhance immersion and usability.
Participants shared several suggestions for improving the next iteration of the prototype to create a more complete and engaging system. They noted that using only two television models was insufficient to fully represent the history of visual media and expressed interest in interacting with a wider range of devices.
To enhance the user interface, participants recommended adding visual or audio feedback to confirm when gestures are correctly performed—such as displaying a message or playing a sound. An onboarding tutorial with clear instructions was also suggested to support users, especially when starting the ‘Learn’ function.
A participant proposed implementing a carousel view for easier access to multiple models without requiring users to physically move around, allowing them to swipe between items instead. It was also recommended to display one 3D object at a time to reduce system load and prevent rendering delays.
Our initial concept envisioned a showcase of television models, each representing a distinct era—past, present, and future. Users would interact with floating UI panels to navigate the system and explore the history and interaction methods of each period. However, this idea came with many problems.
Design & Develop
XRTV - Extended Reality Television
An Interactive Learning System for Exploring Overlooked and Unfamiliar Forms of Interaction with Visual Media
About
Since the 1950s, digital visual media has evolved from the first digital scanner to innovations like television cameras and coloured images. While fewer people use traditional TVs today, the technology continues to advance, including gesture-controlled TVs—though many are unaware of them. This project introduces users to both older and emerging TV technologies through XR (Extended reality) experiences, enabling learning through interactions rather than static text or images.
This project's core goal is to address the lack of contextual understanding surrounding television history - identified from our visit to ACMI [1] - by creating an interactive learning system that facilitates learning through XR technology.
Design Process
Our project followed an iterative design thinking process, the double-diamond method [2]. We began with secondary research to explore possible ways to extend traditional learning methods. We had multiple brainstorming sessions and conducted a literature review to understand television history and XR technology, which guided the direction of our research. Additionally, we conducted a fieldwork study to gain key insights and formed a clearer understanding of participants television consumption, adoption of XR as a learning tool and concluded our first milestone.
Our second milestone started with the design and development phase, where we used different techniques to created different high-fidelity prototypes on ShapesXR and Unity. These prototypes were created con-concurrently and had gone through multiple iterations to achieve a immersive experience. The final milestone focused on user testing when we validated the prototypes design and evaluating the user experience.
Problem Statement
How might we support people in discovering and interacting with overlooked or unfamiliar forms of visual media?
How might we make learning about interface evolution immersive, fun, and accessible even for XR beginners?
Iteration 3 - Hardware Integration
Iteration 4 - Unity Prototype
7
interview participants
10+
questions were asked
Interviews were conducted in a face-to-face setting with a guide to support the interviewer. These interviews provided genuine user insights, capturing their emotions, desires and challenges. Meeting with 7 individuals (with technical and non-technical backgrounds) played a crucial role in shaping our system’s vision.
XR will be helpful for all different disciplines, but it will benefit more from areas that require you to visualise
I think TV remote controls do a good job for their usage, where usually you don't need to input a lot of text
Using XR for education is actually great, but it's expensive, especially for individuals, but looking at the benefits, I think that will be great, because it gives you better immersive experience.
I think XR is easy for users to learn, and you can add more materials inside, and it will make the learning materials more interactive
I prefer voice control and gestures because it offers more autonomy. It automates the process, and it's more of into the device that I just give a voice command and it processes my command and gets stuff done with it
Semi-structured Interviews
Surveys
25
people took the survey
23
questions were asked
Consuming visual media content benefits from intuitive and familiar input methods, particularly for accessibility.
Portable screens are the preferred medium for visual media consumption
XR enhances knowledge visualisation, making learning more accessible
XR technology is underutilised as a learning tool
Gesture control in XR can be integrated with other input methods for a more seamless experience
Physical controls remain more popular than non-physical alternatives.
Survey Insights
After gathering data from both surveys and interviews, we analysed data and used thematic analysis to group and assign codes based on recurrent themes and patterns. Additionally, we used affinity mapping to make sense of large volume of quantitative data.
Affinity Mapping & Thematic Analysis
Finding 1
Users prioritise simplicity and comfort of old technologies over new technologies
Thematic analysis showed that participants rarely use TV for visual media consumption and most were unaware of gesture control technology. 75% of participants preferred traditional remote controls due to their ease of use and familiarity.
For me, a traditional remote control is the easiest to use and a Smart remote control makes me feel comfortable because it's similar to when I type text on a smartphone.
I don't think I have like beyond conducting my own research and sometimes entertainment, I don't think I've used it for learning
Finding 2
Low awareness of adopting XR as a learning tool
Surveys found that nearly 45.8% of respondents were familiar with XR technology, while 41.7% were aware of it but had never used it. Despite this awareness, XR remains underutilised as a learning tool, likely due to insufficient knowledge about its educational applications, limited available content, or the challenging learning curve required to master XR interactions.
Brainstorm
Based on the data from secondary research and fieldwork, we drew 3 sketches to inform our design requirements. Ultimately, we went with sketch 3 due to its tangible and immersive nature
Personas
Storyboard
Low-fidelity Sketches
Mid-fi Prototyping
Based on the data from secondary research and fieldwork, we created two personas, one represent people with technical background and the other is for non-technical individuals. Personas allow for a more-in-depth understanding of the user needs, goals, behaviour and pain points.
Storyboard was used to visually plan how the users would use our system. The storyboard highlights each step the user would take to explore the transition of interactions, from tangible to gesture-control. This approach help us set aside our internal biases and stayed focus on understanding and empathising with user behaviour, keeping us mindful of what drives their actions.
Sketches were created to explore how the system work and visualise the user journey
After we settled on the idea, we did rapid prototyping with ShapesXR to design the UI and Spline for animation to quickly bring our idea to life.
ShapesXR
Spline
Interation 1 - UI & Viewing Direction
Through team discussions, we identified that the horizontal UI panels were too cluttered with text and occupied excessive navigation space, which could negatively impact user engagement. These problems could potentially hinder user engagement. Therefore, we decided to switch to a vertical layout and condense the text on each panel for a clearer and more engaging experience
Initially, we planned to include a third television model, a screenless television that represent the future by connecting a bluetooth keyboard to the VR headset. However, due to time constraint, we were unable to implement this successfully.
One of the major challenges we faced in developing our Unity prototype was working with the hardware components. Throughout the process, the dials on the miniature TV model frequently detached, and several components failed to function as intended, requiring resoldering and hardware adjustments. Even after successfully connecting the miniature model to the Unity software, we encountered noticeable input delays between the physical interactions and their corresponding outputs in Unity.
We initially believed ShapesXR could support more complex interactions, but discovered it is limited to basic digital interactions and does not support MP4 file playback. Since our project involves hardware integration and gesture control, we turned to Unity to implement gesture control, video playback and enable hardware connectivity. Additionally, we later found that ShapesXR does not allow screen sharing during usability evaluations, which presented missed opportunities to collect valuable feedback.
As a result, we used ShapesXR primarily as a design tool to visualise the UI, while Unity was employed to demonstrate and test the user experience.
To evaluate our system, we utilised surveys and usability tests to validate the prototype design and assess the user experience.
We collected a total of 13 survey responses from visitors at the Endeavour Exhibition, following their interaction with our project. These surveys gathered feedback on their experience using the prototype and their overall learning experience through Mixed Reality.
Usability testing was conducted with 12 participants, using moderated observation combined with the Think Aloud method to capture participants’ thoughts and decision-making processes. We observed real-time interactions and collected feedback on usability, functionality, and general user experience. After completing the tasks, each participant answered a set of post-task questions focused on three key areas: clarity of instructions, system engagement, and confidence in navigating the system independently.
Before
Before
Before
After
After
After
Project Setup
System UI
Iteration 2 - Down to 2 television models
High-fi Prototyping
Evaluation Findings
Clarity, visibility, control understanding and system feedback
Engagement and challenges with physical controls
Engagement and challenges with hand gesture controls
Limitations of XR Technology
Learning Experience & Immersion of Extended Reality (XR)
Recommendations for Future Iterations
“I want to see the following...”
It became evident that certain interface elements required refinement to enhance user experience by improving flow and minimising inconvenience.
During the evaluation, users pointed out several issues: the text size was too small, the instruction panel lacked feedback upon task completion, and the system did not provide an instructional overview before interaction—leaving users unsure of where to begin. Additionally, contrary to our initial assumption that a vertical layout would offer a better experience, some users reported neck discomfort and suggested a horizontal layout instead.
Participants generally found the interaction with the physical retro TV model intuitive and engaging, appreciating its nostalgic appeal. However, confusion arose due to the unclear separation between physical and virtual controls, with several users attempting to interact with virtual elements instead of the physical model.
The small size of the model also made it easy to miss, reducing visibility and immersion. Additionally, mismatches between the physical dial positions and virtual outputs caused minor usability issues. Despite these challenges, users remained engaged, highlighting the potential of combining tangible interaction with mixed reality. Future iterations will focus on improving control clarity, model size, and input-output alignment.
Gesture control was found to be more challenging than physical dials, with a clear divide between participants who understood the gestures and those who struggled. While users who successfully performed the gestures found the interaction engaging and enjoyable, others faced difficulties due to inconsistent gesture sensitivity, overlapping gesture recognition (e.g., swipe down vs. push), and lack of clear visual feedback.
Participants were also unsure of the correct hand placement and timing for gestures, as hand positioning guidance was not yet implemented in the Unity prototype. Despite these usability issues—mostly related to software—participants viewed gesture control as a fun and highly interactive feature with strong engagement potential.
While XR offers immersive and engaging interactions, several limitations were observed during usability testing. New users, particularly those unfamiliar with VR headsets, often felt overwhelmed and required additional guidance. Hardware-related issues such as headset weight and cybersickness remain major concerns—causing discomfort, neck strain, dizziness, and nausea, especially with prolonged use.
Additionally, the prototype setup demands a safe and spacious environment. In crowded or constrained areas, headset calibration may be disrupted, and users may struggle to perform necessary gestures or movements. These factors highlight the need for shorter experiences, careful space planning, and user support for effective XR implementation.
The findings support the project’s goal of enhancing learning through extended reality. Participants reported a highly positive overall experience with the prototype, particularly enjoying the comparison between retro and modern TVs. While users were initially more focused on interacting than reading informational panels, engagement increased significantly once they began the ‘Learn’ function. Gesture control was seen as more engaging due to its interactive and playful nature.
Most participants agreed that active, hands-on learning was more effective—especially for children—compared to traditional methods like reading or lectures. Participants also saw potential for XR in other educational contexts, such as safely exploring science experiments. The prototype was considered accessible even to users without prior XR experience, though improvements—such as shortening long text—are needed to further enhance immersion and usability.
Participants shared several suggestions for improving the next iteration of the prototype to create a more complete and engaging system. They noted that using only two television models was insufficient to fully represent the history of visual media and expressed interest in interacting with a wider range of devices.
To enhance the user interface, participants recommended adding visual or audio feedback to confirm when gestures are correctly performed—such as displaying a message or playing a sound. An onboarding tutorial with clear instructions was also suggested to support users, especially when starting the ‘Learn’ function.
A participant proposed implementing a carousel view for easier access to multiple models without requiring users to physically move around, allowing them to swipe between items instead. It was also recommended to display one 3D object at a time to reduce system load and prevent rendering delays.
Actionable tasks
Navigate the learning UI
Give feedback on system UI and navigation flow
Observables
Do users navigate the tasks correctly?
Can they see UIs and the miniature TVs?
Do they request further instructions?
Design Evaluation
Intuitiveness of control: Do the knobs naturally encourage users to interact with them?
Playful discovery: Do the tasks and prototype invite curiosity and exploration without guidance?
User confidence: Do participants feel capable and comfortable navigating the experience independently?
Actionable tasks
Navigate the learning mode on Unity
Turn on/off the flat TV
Gesture controlling the TV (Swipe, push, point)
Observables
Can participants start without verbal help?
Does the system recognise the participant’s hand gestures?
Participants’ struggles while getting familiar with gesture-control interactions
Design Evaluation
Feedback during gestures: Do the participants receive instant feedback while going through the task
Task Design - Retro TV
Task Design - Gesture-control TV
Secondary Research
Primary Research
7 Interviews + 24 Surveys
To better understand the gap between user needs and past research, we conducted interviews and survey questionnaires. Participants were asked about their visual media consumption and their general knowledge of XR technologies.
All interview participants provided informed consent via a consent form that allowed their audio recordings to be transcribed and analysed.
Younger generations increasingly using smartphones and social media instead of conventional television
Younger people are unfamiliar with older models
XR in visual media education is still limited
XR-based learning enhances motivation and learning outcomes more effective than traditional learning methods
XR technologies offer immersive learning experiences, making them powerful tools for engagement and retention
Gesture-control television are intuitive and engaging but also present learning challenges
Literature Review
Media consumption has changed
Television Literacy
XR challenges with integrated audio-visual cues
XR-based learning
XR technologies
Gesture-based television
Semi-structured interviews
7 participants
Facet-to-face
Interview guide
Surveys
24 respondents
Closed and open-ended questions
Survey questionnaires
Conclusion
References
Our proposed system is grounded in user research and developed through two prototypes, each individually emphasising on user interface and user experience. The evaluation results underscore the educational benefits of interactive and immersive learning, especially for young children. A major challenge in this project has been designing for XR, which requires seamlessly integrating the interface with both human interaction and real-world objects and environments.
Looking ahead, we plan to refine the system and enhance hardware integration to build a more comprehensive learning environment. This includes conducting interviews and cognitive walkthroughs with experts in education and emerging technologies to deepen our understanding and improve the project. Due to time constraints, only two television models were included in the current prototype; future versions should feature a broader historical range, incorporate audio and visual feedback, and expand the selection of TV models to better illustrate the evolution of visual media.
[1]https://www.acmi.net.au/education/school-program-and-resources/texts-stories-and-culture-australian-tv-miniatures/
[2]https://www.hi-interactive.com/blog/ux-process-the-double-diamond
Solution
An Interactive Learning System for Exploring Overlooked and Unfamiliar Forms of Interaction with Visual Media
During our fieldwork study, we identified that people nowadays rarely use televisions to consume visual media and show low awareness or adoption of gesture-based controls and XR technologies, especially as learning tools. Our project aims to bridge this gap, by expose users to unfamiliar technology like gesture-based control television and older technology with a showcase of several digitalised television models using Meta Quest 3 VR headset.
ShapesXR prototype for showing the onboarding instructions and the system’s UI
ShapesXR was utilised to design the user interface, offering a clear visual representation of the system’s structure and functionality. It also enabled the creation of immersive experiences by integrating 3D objects and spatial interactions, allowing users to engage with the content in a more intuitive and interactive manner.
To achieve more a realistic experience, Unity was utilised to replicate real TVs interactions. Using the same TV models from ShapesXR, this prototype aims to bridge the gap between the physical and virtual worlds. We began by acquiring miniature TV models and identifying the necessary hardware components for the project.
Through iterative development and integration of the Meta Quest 3 toolkit and its Pass Through feature, we built a mixed reality system that have both tangible and virtual interactions—allowing users to change channels and adjust volume using a physical dial on a miniature retro TV while seeing these changes reflected on a virtual digital twin.
In addition, we successfully implemented gesture-controlled functionality, enabling users to operate the TV using hand gestures.
Through guided instructions via the UI panels and virtual buttons, users can navigate and gain a better understanding of how old television models works
Moving on to a more present television model, users will get to test gesture-control technology that offers a unique experience to control TV at the wave of their hands
Unity prototype replicating the interactivity and realistic experience of using TVs
2 Prototypes
ShapesXR Prototype
Unity Prototype
Part 1 - Retro TV Interactions
Part 2 - Gesture-control TV Interactions
User Evaluation Methods
Esp 32
Breadboard
3D Printed Knobs