Ann Li

Hand-written name, "Ann Li"

Ann Li

is a multidisciplinary interaction designer exploring experiential narratives and creative tech

Best viewed on desktop

Ann Li

Neurobiologist-turned designer driven by storytelling, sense-making, and curiosity. Seeking full-time interaction, experience, or product design roles in the US.

Work spans the digital and physical, including interactive exhibits, creative direction and strategy, and functional user interfaces.

Augmenting Embodied Learning

Integrating XR and tinyML to train skilled workers for future opportunities in creative arts and manufacturing

Roles

Design Research  
Workshop Facilitation
Feature Development
Interface Design
XR Prototyping

Context

Jan 2023-May 2024

Team

Zhenfang Chen
Andrew Knowles
Tate Johnson
Semina Yi
Yumeng Zhuang
Dina El-Zanfaly
Daragh Byrne

Overview
Outcomes
ResearchApproachDesignReflections

Overview

Despite the prevalence of welding in construction, manufacturing, and creative arts, training novice welders is a time-intensive and challenging process.

Metal welding is a craft manufacturing skill that can be unusually difficult to externalize and represent to novices. Building competency requires an apprentice to iteratively practice embodied skills and sensitize themselves to a sensorially complex practice. Learners must build deep embodied knowledge such as hand-eye coordination, muscle memory, and sound identification.

To explore these challenges, a series of co-design workshops were conducted with a youth program in welding and fabrication. Opportunities for mixed reality, sensing, and tinyML processes to augment welding training and practice were identified, leading to the design of an extended reality welding helmet and torch.

Outcomes

An XR system assists with limitations of traditional welding education, by providing realtime, in-helmet, and in-situ feedback to augment the educational experience.

By enabling users to encounter real behaviors, forces, sounds, and overall experience of an active weld, we provide a novel in-situ experience with spatial and biometric analysis.

The designed system allows us to explore other aspects of embodied learning in welding practice, including the analysis of biometric data, performance analysis, and the inclusion of meditation to further augment the educational experience.

Modified welding helmet and gun

We apply machine learning to a lightly-modified off-the-shelf XR and welding setup to enable in-situ welding training, enhancing the embodied learning of welding in three key ways.

  1. Visual XR Guides and Integrated Motion Sensing. Combined motion-sensing and visual XR feedback helps improve proprioceptive and embodied learning.
  2. Sensing Sonic Cues During Welding Practice. Acoustic sensing focuses learner attention on non-visual cues of weld performance.
  3. Mediated Meditation and Regulation. Biometric sensing enhances mindfulness and stress management in sensorially challenging environments.
Simplified system diagram

Community Workshops

We worked closely with the Industrial Arts Workshop (IAW), a non-profit youth welding training program to inform our approach, which was grounded in co- and participatory design. A series of three on-site community workshops were conducted to explore welding instructor and student needs, achieve working understanding of learning contexts, and identify design opportunities along key moments of the curriculum.

Focus groups with instructors and leadership

Immersive, activity-based engagement

Participating in welding training immersion

We worked with stakeholders (instructors and students) in their context of use, making several visits to the Industrial Arts Workshop (IAW) site, where we experienced firsthand the challenges, highlights, goals, and considerations encountered by novice welders.

I led the design and facilitation of a series of co-design workshops and participatory ideation sessions. It was important to us that we emphasize a participatory design approach, rather than classical theory-based or researcher-led approaches. In order to accomplish this, I aimed to identify and provide suitable tools and processes for our community partners. I developed workshop agendas that incorporated various methods, including projective techniques (diagramming, journey mapping, storyboarding), and creative activities.

Experiential co-design workshops conducted with community collaborators at the partner site

Designing new methods that gain creative input and trust

However, IAW partners were not familiar with XR interfaces, and tools and techniques for co-designing mixed reality experiences are limited in adaptability and transferability. As a result, we opted to focus on storyboarding and scenario making as an accessible means to co-design. This also resulted in the design of novel creative toolkits for participatory prototyping of XR interfaces, an accessible, flexible, and intuitive medium to solicit design input and feedback on features.

Participatory co-design of HUD interface
Accessible prototyping for XR visual interfaces

Novel Contributions

The novelty of our work includes the ability for it to be used as an in-situ, functioning tool, as well as our holistic approach to augmentation.

Prior work in the areas of extended reality and welding create isolated, fully immersive simulations of welding, rather than an augmentation of actual welding in context. Existing products and research include full VR welding experiences (training without a functional helmet or torch, but a visual augmentation with hand tracking) and AR/computer vision-based simulations using tracking markers.

Our system is implemented to work with a live spark, in real welding contexts. Beyond training muscle memory, our system also uses additional sensors and the Quest's onboard sensors to promote good practice through feedback on mindful breathing, weld performance, and other attributes

‍

Device assemblies

‍

‍

Publications and Press

Details on our co-design process and hardware development, system setup, training specifics, related works, and citations

Augmenting Embodied Learning in Welding Training: The Co-Design of an XR- and tinyML-Enabled Welding System for Creative Arts and Manufacturing Training
‍ACM Tangible Embedded and Embodied Interaction 2024
Honorable Mention

Augmenting Welding Training: An XR Platform to Foster Muscle Memory and Mindfulness for Skills Development
‍ACM Interactive Surfaces and Spaces 2023
Best Demo

XRweld: An In-Situ Extended Reality Platform for Welding Education
ACM SIGGRAPH 2024

Machine learning and extended reality used to train welders
Feature: Carnegie Mellon University College of Engineering

Build Back Better Grant

Click to navigate to the top
A Changing Arctic, Carnegie Museum of Natural History
Augmenting Embodied Learning
Cara, CVS Pharmacy
Wishwell
Artwork
Music Programming
Hand-written name, "Ann Li"
InstagramTwitterLinkedinBehanceDribbble

Ann Li 2020

Don't be a stranger
If you think I'd be a good fit for your project or team, let's connect.
© 2023 Ann Li
LinkedIn link