AI has been ingrained in the sports and fitness industries for a few years already. Since computer vision has become more advanced and available both for athletes and entrepreneurs, we can see the emergence of a new subset of applications.

AI virtual coach or application that assists athletes in training, correction, analysis of exercises, as well as rehabilitation procedures. Whether this is true or just another hyping technology, I'll elaborate in this article using cases from my practice.

Human Pose Estimation Virtual Coach in a Smartphone

The leap spread of HPE technologies in sports and fitness has been due to the progress of computer vision. Today, movement tracking does not require the mandatory use of additional smart wearables, as it was until recently. The current level of computer vision allows accurate real-time tracking of fast movements.

Hence, it is possible to perform such a computer vision task as Human pose estimation (HPE) with just an optical sensor and a good amount of light. In this way, HPE has become another technology that can be accessed with the camera of a smartphone. Human pose estimation is aimed at recognizing and accurately tracking certain points on the human body, for example, limbs and joints, facial points, or even fingers. Based on these points, we can identify motion patterns, specific joint positions, and body poses. Analysis of motion patterns and poses, in turn, enables us to make data-driven decisions about their estimation and possible correction during exercise, professional coaching, or even sports events.

Accurate assessment of exercise technique, substantiated corrective feedback for athletes, and comparison with perfect biomechanics of exercise performance—this is what is brought to AI software products in the sports industry with the implementation of HPE technologies. The more accurate models for tracking motion emerge, the more applications for professional sports enter the market. 

The Basics of Human Pose Estimation

Mostly, the application of HPE is based on the recording of RGB images to detect body parts and track their movement in 3D space. The raw data collected in this way is uploaded to the HPE system for processing. Two-dimensional selection of a point takes into account only 2D space and does not allow for perceived depth and, therefore, has poor accuracy. 

The use of 3D pose estimation models allows for greater tracking accuracy, albeit under decent lighting conditions. In practice, it is advisable to use both approaches in combination because with the 2D method, you can quickly identify the actual key points, and with the 3D method, you can achieve the required accuracy and correct perception of space.

Detecting and extracting key points from a sequence of images are the two main parts of the human pose estimation flow. Using horizontal and vertical coordinates, you can create a skeleton structure. The HPE model analyzes every frame and detects key points in the human body. Processing of the sequence of frames occurs simultaneously with the consideration of past data to provide more accurate pose estimation.

Of course, in technology, as in sports, nothing comes without effort. Our AI engineers overcome the shortcomings of existing ready-made detection of 3D human pose solutions by applying a speed threshold correction method for each joint, correction of some values using statistics, additional filtering of joint data, and other technical tricks. However, in my opinion, Mediapipe solutions, reinforced by their application expertise, provide the necessary pose recognition accuracy for coaching apps while being more affordable than developing HPE models from scratch.

What are the capabilities of an AI virtual coach?

Of course, the capacity to analyze human motion patterns is not limitless today. The correctness of video recording is still essential to get decent results. In particular, any human pose estimation app requires proper lighting, shot angles, and user outfits that do not blend with the environment or hide the detection key points. 

Although once used properly, a virtual coach application can serve its user with a vast spectrum of coaching assistance. By analyzing HPE outputs in conjunction with athletic data and biomechanical principles, the virtual coach can:

• Identify and analyze the sports technique flaws. Comparing the user's movements to exemplary, perfect exercises reveals deviations and errors.
• Provide instant personalized feedback. A mentor from a smartphone belongs entirely to the user in contrast to a human coach, who can scatter his attention between several athletes working out at the same time. An HPE-based app provides real-time, individualized feedback tailored to a specific athlete's strengths and weaknesses.
• Design customized training programs. The AI model can leverage data about movement patterns, user metrics, and objectives to create personalized training plans.
• Track and measure progress. The HPE model can quantify improvements and modify training by continually monitoring the athlete's performance.

As any AI-based approach requires significant amounts of data to train on, Human Pose Estimation may seem like the prerogative of businesses that spend years collecting and aggregating data. However, I can surely tell there are successful examples of similar projects that started from the ground up. BeOne Sports represents a growing platform that implements HPE as its pivotal feature to enable AI-powered training for professional athletes. 

Since MobiDev takes care of app development for this platform, I have an insider's view of how BeOne Sports empowered athletes by allowing them to compare their movements with those of renowned athletes. Before I share the highlights of this product, let's review how Human Pose Estimation works from a technical perspective.

Sports Performance Analytics and Corrective Feedback with AI

Data quality determines the quality of the AI app. If the technical limitations of video data collection can be overcome through disciplined and skilled video recording, then the issue of data availability requires a more creative approach.

Technologies of human estimation of poses require vast amounts of qualitative data specific to a task. This means tons of golf exercises recorded and formed in a dataset would be irrelevant to working with basketball movements or wrestling. It is necessary to take into account the variability of the same exercises, the peculiarities of the human body, and much more. Data obtained from athletes whose effectiveness is generally recognized and confirmed by sports results are of particular value.

The recipe for the success of the BeONE Sports platform was the involvement of international-level athletes, whose sports technique is considered to be a benchmark. Due to the implementation of this creative idea, BeONE Sports obtained the required sets of high-quality video data in athletics, football, baseball, volleyball, and basketball and created a data collection and preparation system suitable for many other sports.

Thus, BeONE Sports and MobiDev have provided athletes with an AI-powered comparative training platform to help them rise to the next level. Just as a championship jump consists of separate elements, the interaction of athletes with the BeONE Sports platform consists of several clear steps:

• Athletes record a specific exercise using the BeOne Sports application.
• The HPE-powered app identifies key body points and tracks movement patterns.
• The app compares the users' movements to perfect repetitions of the same movements of the top athletes, the videos of which are gathered in a custom database.
• Users receive real-time feedback on their form and the correctness of the exercises. By assessing the athletes' technique and form, the virtual coach simultaneously provides them with personalized training recommendations highlighting zones that need improvement.
• Users can now visualize the exercise technique using graphs for specific parameters like knee height, hip position, trajectory of movement, etc.

The given AI-powered comparative training platform case shows the possibilities of applying HPE technologies in sports and fitness and, at the same time, proves that they are far from exhausted. 

Since the HPE Model is a core element of a virtual coach, I emphasize that, most often, there is no need to custom-develop it from scratch. For example, our company often uses MediaPipe to accelerate and reduce the cost of developing AI-enhanced apps.

The MediaPipe open-source project provides both the MediaPipe Framework, the low-level component to build efficient on-device machine learning pipelines, and MediaPipe Solutions, a suite of libraries and tools to apply artificial intelligence (AI) techniques in apps quickly. 

This project also includes identifying body locations, analyzing poses, and categorizing movements using machine learning (ML) models that work with video. Among the MediaPipe Solutions are pre-trained, ready-to-run MediaPipe Models, as well as cross-platform APIs and libraries for deploying software products. For us, as experts in the development of AI-powered apps, the possibilities of customizing ready-made solutions are essential. In the case of MediaPipe, the relevant toolkit consists of MediaPipe Model Maker for customizing models with your own data and MediaPipe Studio for visualization, evaluation, and benchmark solutions.

Further development of HPE technology will allow, in the near future, the creation of virtual coaches who will be able to not only analyze the movements of athletes but also understand their physical condition and level of fatigue. Such AI assistants will offer holistic guidance to optimize physical and mental performance for fitness enthusiasts. I see a significant benefit in implementing features of not only sports training but also health and physical rehabilitation in one app. The possibilities of applying the achievements of different subfields of artificial intelligence in one software product are also becoming stronger. Personalized training recommendations, individual training programs, universal trainer bots, advice on injury prevention, and physical rehabilitation will become integral elements of mobile fitness apps.

Fitness and Sports Apps with HPE: Through Data to Improvement

The similarity between sports and AI is that both are ever-evolving. Fitness constantly receives powerful impulses for development from artificial intelligence technologies. Platforms for athletes and coaches based on human pose estimation (HPE) are evolving in a spiral. The popularity of such software products grows due to the groundbreaking feature of comparing the correctness of exercise performance. This trend, in turn, contributes to the involvement of even more leading athletes, who provide videos of their training for the app. Thus, more high-quality data becomes available, which helps iteratively improve the app while increasing the range of sports it covers. 

What could be more comforting for product owners than watching the simultaneous progress of their app and the sports results of its users? Earning the devotion of sports enthusiasts due to the unprecedented accessibility of coaching, acceleration of performance enhancement, user-friendliness, and barrier-free training, HPE-powered fitness applications have an exciting future.