Chen-Chieh (Jacky) Liao

Understanding neuromusculoskeletal mechanisms significantly impacts skill specialization and proficiency. While existing methods can infer large muscle activities during gross motor movements, the estimation of dexterous motor control involving miniature muscles remains underexplored. Targeting the coordinated hand muscles in advanced piano performance, we learn spatiotemporal discrete representations of electromyography (EMG) data and hand postures utilizing a multimodal dataset. Subsequently, we train a precise and cost-effective neural network model. Based on this model, PiaMuscle is introduced to investigate if visualizing muscle activities during piano training enhances piano performance. Quantitative and qualitative results of a user study with highly skilled professional pianists demonstrate that PiaMuscle provides reliable muscle activation data to support and optimize force control. Our research underscores the potential of a naturalistic workflow to estimate small muscles’ activities from readily accessible human-centric information and more accurately when combined with tool-centric data, thereby enhancing skill acquisition.

Sports performance is often hindered by unintentional habits, particularly in golf, where achieving a consistent and correct swing is crucial yet challenging due to ingrained swing path habits. This study explores redirection approaches in virtual reality (VR) to correct golfers’ swing paths through strategic ball shifting. By initiating a forward ball shift just before impact, we aim to prompt golfers to react and modify their swing motion, thereby eliminating undesirable swing habits. Building on recent research, our VR-based methods incorporate a gradual transformation of visuomotor associations to enhance motor skill learning. In this study, we develop three ball shift patterns, including a novel pattern that employs gradual ball shifts with interspersed normal conditions, designed to retain learning effects post-training. A preliminary study, including expert interviews, assesses the feasibility of various ball-shifting directions. Subsequently, a comprehensive user study measures the learning effects across different ball shift modes. The results indicate that our proposed redirection mode effectively corrects swing paths and yields a sustained learning effect.

Egocentric 3D human pose estimation, which estimates an individual’s 3D pose from a camera attached to a part of their body, operates within a specialized camera domain to capture the entire body. This specialization makes it challenging to collect real-world training data due to the difficulty in acquiring diverse and accurately labeled data from the egocentric perspective. Consequently, most existing methods rely on synthetic data for training, which increases the mean joint error when applied to real-world images due to the domain gap. Some works have addressed this issue by generating pseudo-labels from synchronized real egocentric and exocentric images and using them for training. However, this approach is costly in terms of data collection, making it difficult to scale and apply to other camera setups. In this work, we propose a novel method that employs a diffusion model with ControlNet to generate real-world-like images, thereby reducing the domain gap. The proposed method relies on synthetic data, which is easy to acquire, and a small amount of text-captioned real-world data. This method easily applies to estimating egocentric 3D human poses across various camera setups. Experiments with two different camera setups demonstrated that models trained with images generated by the proposed method improve accuracy with real-world data. Specifically, the PA-MPJPE of the Mo2Cap2 model improved by 8.9% in the SceneEgo test set and by 3.4% in the GlobalEgoMocap test set.

Controlling vocal cord muscles is crucial for vocal performance and training, yet it is challenging to measure. This paper introduces electromyography (EMG) and ultrasonography to detect vocal muscle activity and assess pitch training skills. A pre-experiment with 16 participants analyzed muscle control discrepancies among singers of different skill levels. A subsequent user study with 12 participants evaluated EMG and ultrasonography feedback effectiveness. Findings indicate that EMG offers better temporal stability representation, while ultrasonography provides intuitive visual feedback on vocal cord activity. Both methods show potential in enhancing vocal control, offering insights for designing effective and non-invasive vocal training systems.

Alpine skiing is a popular winter sport, and several systems have been proposed to enhance training and improve efficiency. However, many existing systems rely on simulation-based environments, which suffer from drawbacks such as a gap between real skiing and the lack of body ownership. To address these limitations, we present ARpenSki, a novel augmented reality (AR) ski training system that employs a see-through head mounted display (HMD) to deliver augmented visual training cues that may be applied on real slopes. The proposed AR system provides a transparent view of the lower half of the field of vision, where we implemented three different AR-based direct and indirect postural visualization methods. We conducted an user study to investigate the influence of different visual cues in the AR environment. Our results indicate that a simple AR visualization of the user’s spine (Figure 1.2) yields the most favorable training performance, surpassing conventional visualizations by 7% improvement in the user’s posture. Building upon these promising findings, we further tested our system on real slopes and showed the potential of a real AR skiing application.

Effective analysis of skills requires high-quality, multi-modal datasets, especially in the field of artificial intelligence. However, creating such datasets for extreme sports, such as alpine skiing, can be challenging due to environmental constraints. Optical and wearable sensors may not perform optimally under diverse lighting, weather, and terrain conditions. To address these challenges, we present a comprehensive skiing/snowboarding dataset using a professional motor-based simulator. Using the realistic simulator, it is easy to obtain different types of data with a small domain gap between real-world data. Common data for skill analysis are collected, including camera images, 3D body pose, sole pressure, and leg electromyography, from athletes of different levels. Another key aspect is the comparison of cross-modal baselines, highlighting the versatility of the data across modalities. In addition, a real-world pilot test is conducted to assess the practical applicability and data robustness.

Motor skill acquisition involves learning from spatiotemporal discrepancies between target and self-generated motions. However, in dexterous skills with numerous degrees of freedom, understanding and correcting these motor errors are challenging. This issue becomes crucial for experienced individuals who seek for mastering and sophisticating their skills, where even subtle errors need to be minimized. To enable efficient optimization of body posture in piano learning, we present PianoSyncAR, an augmented reality system that superimposes the time-varying complex hand postures of a teacher over the hand of a learner. Through a user study with 12 pianists, we demonstrate several advantages of the proposed system over conventional tablet-screen, which implicate the potential of AR training as a complementary tool for video-based skill learning in piano playing.

This work proposes a motion style transfer network that transfers motion style between different motion categories using variational autoencoders. The proposed network effectively transfers style among various motion categories and can create stylized motion unseen in the dataset. The network contains a content-conditioned module to preserve the characteristic of the content motion, which is important for real applications. We implement the network with variational autoencoders, which enable us to control the intensity of the style and mix different styles to enrich the motion diversity.

Creating the photo-realistic version of people’s sketched portraits is useful to various entertainment purposes. Existing studies only generate portraits in the 2D plane with fixed views, making the results less vivid. In this paper, we present Stereoscopic Simplified Sketch-to-Portrait (SSSP), which explores the possibility of creating Stereoscopic 3D-aware portraits from simple contour sketches by involving 3D generative models. Our key insight is to design sketch-aware constraints that can fully exploit the prior knowledge of a tri-plane-based 3D-aware generative model. Specifically, our designed region-aware volume rendering strategy and global consistency constraint further enhance detail correspondences during sketch encoding. Moreover, in order to facilitate the usage of layman users, we propose a Contour-to-Sketch module with vector quantized representations, so that easily drawn contours can directly guide the generation of 3D portraits. Extensive comparisons show that our method generates high-quality results that match the sketch. Our usability study verifies that our system is preferred by users.

Video-based lessons are becoming a popular way for distance piano education. However, limited by the fixed camera angle, a video is difficult to tell precise 3D hand posture, which is one of the most essential factors for learning piano. This paper presents a visualization system providing the intuitive discrepancy of hand postures in two piano performance videos. Through a motion capture system, the estimated 3D postures are visualized and discrepancies based on distinct metrics are displayed, integrated with modular functions assisting skill acquisition. A pilot study proves that the proposed visualization can be a supplementary means for only video-based lessons in terms of correcting hand postures and fingering.

Spatial and temporal clues found in a professional’s motion are essential for designing a training system for learning a motor skill. We investigate the potential of using neural networks to learn spatial and temporal features of advanced players in sports and to detect the fine-grained differences between motions. As a training system, we implement an AI Coach prototype application that finds the differences between two input motions and visualizes a recommendation motion for the users to correct their forms. In the user study, we investigate the effects of the proposed AI Coach and discuss the findings based on quantitative questionnaires and qualitative interviews. In the study, the proposed system can help the user better understand the difference between them and the coach. The study also reveals the necessity of coaching beginners in the early learning phases.

In the field of the acquisition of sports skills, a common way to improve sports skills, such as golf swings, is to imitate professional players’ motions. However, it is difficult for beginners to specify the keyframes on which they should focus and which part of the body they should correct because of inconsistent timing and lack of knowledge. In this study, a golf swing analysis tool using neural networks is proposed to address this gap. The proposed system compares two motion sequences and specifies keyframes in which significant differences can be observed between the two motions. In addition, the system helps users intuitively understand the differences between themselves and professional players by using interpretable clues. The main challenge of this study is to target the fine-grained differences between users and professionals that can be used for self-training. Moreover, the significance of the proposed approach is the use of an unsupervised learning method without prior knowledge and labeled data, which will benefit future applications and research in other sports and skill training processes. In our approach, neural networks are first used to create a motion synchronizer to align motions with different phases and timing. Next, a motion discrepancy detector is implemented to find fine-grained differences between motions in latent spaces that are learned by the networks. Furthermore, we consider that learning intermediate motions may be feasible for beginners because, in this way, they can gradually change their pose to match the ideal form. Therefore, based on the synchronization and discrepancy detection results, we utilize a decoder to restore the intermediate human poses between two motions from the latent space. Finally, we suggest possible applications for analyzing and visualizing the discrepancy between the two input motions and interacting with the users. With the proposed application, users can easily understand the differences between their motions and those of various experts during self-training and learn how to improve their motions.

In sports like golf, the motion of a club or racket determines the outcome significantly, and it is essential to consistently move such properties with the correct forms to achieve high scores during competitions. This paper proposes a novel projection method that provides real-time visualization of golf club trajectories on the ground. The system can render the trajectory of a golf club along with conventional face-to-path angle information in real-time using an optical motion capture system and a video projector. Furthermore, the system utilizes dynamic changes of color to provide more detailed information about the pose of the golf club. This color-changing method aims to help users understand their club motion trajectories and explicitly adjust their form based on the visualization results.

Using smartphones while walking is becoming a social problem. Recent works try to support this issue by different warning systems. However, most only focus on detecting obstacles, without considering the risk to the user. In this paper, we propose a deep learning-based context-aware risk prediction system using a built-in camera on smartphones, aiming to notify ”smombies” by a risk-degree based algorithm. The proposed system both estimates the risk degree of a potential obstacle and the user’s status, which can also be used for distracted driving or visually impaired people.

When learning a dexterous skill such as playing the piano, people commonly watch videos of a teacher. However, this conventional way has some downsides such as limited information to be retrieved and less intuitive instructions. We propose a virtual training system by visualizing differences between hands to provide intuitive feedback for skill acquisition. After synchronizing the data, two visual cues are proposed including a hand-overlay manner and a two-keyboards visualization. A pilot study confirm the superiority of the proposed methods over conventional video-viewing.

In this work, we present an analysis tool to help golf beginners compare their swing motion to the swing motion of experts. The proposed application synchronizes videos with different swing phase timings using the latent features extracted by a neural network-based encoder and detects key frames where discrepant motions occur. We visualize synchronized image frames and 3D poses that help users recognize the differences and key factors that can be important for their swing skill improvement.

In order to improve the performance in putting, we design a weight switching device that can provide various switching angles. This paper proposes a system that can change the center of mass by switching the weight to different positions around the head of a putter. The proposed system starts switching the weight at the beginning of a downswing and ends before the putter hits a ball. To verify the effectiveness of the proposed system, we conducted a user study and examined if the face to path angle of the putter changed when the weight was switched to different positions. In the user study, the participant performed putting with different switching angles. In the analysis, we focused on the differences in the face to path angle among the switching angles. The user study showed that the proposed system is effective in changing the face to path angle when switching the weight away from the putter’s shaft. Based on the experimental results, the proposed system contributes to affecting the face to path angle dynamically in real-time.