Motion prediction using brain waves based on artificial intelligence deep learning recurrent neural network
Article information
Abstract
Electroencephalogram (EEG) research has gained widespread use in various research domains due to its valuable insights into human body movements. In this study, we investigated the optimization of motion discrimination prediction by employing an artificial intelligence deep learning recurrent neural network (gated recurrent unit, GRU) on unique EEG data generated from specific movement types among EEG signals. The experiment involved participants categorized into five difficulty levels of postural control, targeting gymnasts in their twenties and college students majoring in physical education (n=10). Machine learning techniques were applied to extract brain-motor patterns from the collected EEG data, which consisted of 32 channels. The EEG data underwent spectrum analysis using fast Fourier transform conversion, and the GRU model network was utilized for machine learning on each EEG frequency domain, thereby improving the performance index of the learning operation process. Through the development of the GRU network algorithm, the performance index achieved up to a 15.92% improvement compared to the accuracy of existing models, resulting in motion recognition accuracy ranging from a minimum of 94.67% to a maximum of 99.15% between actual and predicted values. These optimization outcomes are attributed to the enhanced accuracy and cost function of the GRU network algorithm’s hidden layers. By implementing motion identification optimization based on artificial intelligence machine learning results from EEG signals, this study contributes to the emerging field of exercise rehabilitation, presenting an innovative paradigm that reveals the interconnectedness between the brain and the science of exercise.
INTRODUCTION
The human brain plays a central role in coordinating body movements by processing sensory information and generating motor output signals (Chen et al., 2022; Ratey and Loehr, 2011). The level of brain-muscle tension, which correlates with action potentials’ frequency and amplitude, is indicative of human movement intentions (Yoo, 2020). Electroencephalogram (EEG) research plays a crucial role in gaining insights into human body movements. However, conventional quantified EEG techniques have limitations, including subjective interpretation errors and challenges in visual analysis. These techniques lack the ability to differentiate between normal and abnormal patterns and rely solely on statistical deviations. To overcome these limitations, the development and application of AI-based research tools offer promising solutions (Liao et al., 2020). Previous studies have explored various aspects of human brain wave analysis using AI, with a focus on clinical pathology areas such as epileptic seizure detection, senile dementia, mental disorders, and alcohol addiction (Eun et al., 2021; Gemein et al., 2020; Hamet and Tremblay, 2017). Additionally, EEG has been utilized to examine psychological states, including emotion recognition, academic performance, and rehabilitation training (Chen et al., 2022; Wang et al., 2014). The fields of brain-computer connection and brain-machine interface have facilitated communication between humans and computers using EEG data, particularly in rehabilitation training (Daly and Wolpaw, 2008; Wolpaw et al., 2002).
Machine learning, particularly deep learning and system-based learning, has been widely applied in previous research cases (Adege et al., 2020; Chen et al., 2022; Eun et al., 2021; Liao et al., 2020). The gated recurrent unit (GRU) algorithm, a deep learning technique, has gained attention in the field of artificial intelligence (AI). It leverages machine learning to correct errors between input signal data and expected output. Among the various deep learning recurrent neural networks, the long short-term memory algorithm has been commonly used to mitigate the problem of loss effects in recurrent neural networks for short-term memory tasks. However, due to its complexity and computational demands, the GRU technology employed in this study has been identified as the most suitable model (Liao et al., 2020).
The GRU algorithm has been successfully applied in various contexts. The GRU algorithm has also shown promise in assessing location tracking and trajectory models, predicting the mobility of Internet of things (IoT) device users. There is a growing demand for the development and application of AI technology in brain wave research to explore the connection between the brain and movement. Jeon et al. (2019) achieved up to 92% accuracy in automatically classifying sleep stages using brain wave data, while Liao et al. (2020) utilized the GRU algorithm to predict cybersickness with an impressive 98.88% accuracy, surpassing traditional questionnaire methods. Furthermore, the GRU algorithm has been employed in the assessment of location tracking and trajectory models, effectively predicting the mobility of IoT device users (Adege et al., 2020). However, no previous research has employed the GRU algorithm to predict movement patterns by learning the relationship between brain signals and movement types using EEG data, highlighting the innovative potential of this approach.
This study utilizes the GRU deep learning algorithm, previously employed in related research, to classify human body motions based on exercise types, taking into account the information conveyed by the brain’s exercise command signals (Riccio and Stoffregen, 1988; Yoo, 2020). The main focus is on optimizing performance indicators to improve the accuracy of motion prediction. Through these efforts, this study aims to contribute to the latest findings in posture control prediction and movement identification based on EEG information. Building upon previous studies by Yoo (2019) and Yoo (2020), this paper focuses on the application of the GRU model in studying the correlation between brain signals and movement. Specifically, it investigates the optimization of performance indicators to enhance the accuracy of motion prediction. The ability to maintain an upright posture is crucial for evaluating the health of the central nervous system and serves as an important indicator of potential cranial nervous system disorders. By optimizing an AI machine learning-based deep learning recurrent neural network prediction algorithm, this study aims to provide novel research outcomes in the field of health and rehabilitation.
MATERIALS AND METHODS
In this study, EEG data from previous studies (Yoo, 2019; Yoo, 2020) was utilized. The following provides a concise overview of the experimental information.
Experimental participants and methods
The participants in the experiment were healthy males in their early 20s with no history of brain disease or neuromuscular disorders within the past 6 months. The participant pool (n=10) consisted of elite gymnasts (n=5, specializing in floor exercise, iron bar, jumping box, ring, and vault) as well as college students majoring in physical education (n=5). These individuals performed five different exercise patterns associated with upright postural controls. The first pattern involved sitting on a chair (type 1; class 0), the second pattern entailed standing on double legs (type 2; class 1), and the third pattern focused on single-leg standing. The single-leg standing posture was further categorized into three types: dominant single-leg standing (type 3; class 2), nonominant single-leg standing (type 4; class 3), and dominant single-leg standing with closed eyes (type 5; class 4). Posture measurements were conducted individually for a duration of 1 min, with each posture being repeated twice. The sitting posture required maintaining an upright position while keeping the buttocks in close contact with the chair support. For the double-leg standing posture, participants were instructed to align their head, shoulders, pelvis, and the center line of their feet in a vertical arrangement. During the single-leg standing posture, participants extended their arms while ensuring that the entire sole of the supporting leg remained in contact with the ground, and the thigh of the opposite leg stayed parallel to the ground. Measurements were taken for both the left and right legs during the one-legged standing posture. When participants stood on their right foot, it served as the dominant single-leg stance, while the left foot acted as the nondominant stance. During the single-leg standing with eyes open, participants were instructed to focus on a marker positioned at a height of 1.5–1.8 m, placed 5 m away at 10°–15°. Conversely, during the single-leg standing with eyes closed, participants wore an eyepatch to block visual information (Yoo, 2019; Yoo, 2020).
EEG raw-data collection
The EEG data collected shows potential fluctuations in the frequency range of approximately 1–60 Hz and a potential fluctuation magnitude of 5–300 μV. Various factors contribute to changes in brain waves, including individual differences, age, alterations in consciousness, mental and perceptual activities, physiological changes in the body, and pathological processes or brain-related diseases. This study utilizes the Neuronics32 Plus, a state-of-the-art digital EEG system, to capture electrical activity signals from the brain. The system categorizes the four types of brain waves into four specific brain locations and indicates whether they are hyperactive, depressed, or within the normal range. The EEG measurement setup consists of a 32-channel EEG panel that displays the electrode locations during EEG recording, and the specific names and positions of the electrodes for each brain region are illustrated in Fig. 1. The primary areas targeted for EEG measurement are the frontal lobe (10 channels: FP1, FP2, F7, C3, CZ, C4, CP3, CPZ, CP4, P3, PZ, P4), occipital lobe (3 channels: O1, OZ, O2), and earlobe electrodes (A1, A2) (refer to Fig. 1). Within the Neuronics32 Plus system, the power values for each frequency band of the 32-channel EEG voltage are obtained through spectral analysis (fast Fourier transform), which involves performing a Fourier transform of the recorded EEG signals in the time domain to analyze the frequency domain. The frequency band and amplitude are then calculated based on the amplitude values and frequency periods. The EEG activation potentials observed during exercise are assessed to determine the delta waves (0–3 Hz), theta waves (4–7 Hz), alpha waves (8–13 Hz), and beta waves (over 14 Hz) within each frequency band. During EEG recording, the automatic filtering function applies a noise removal technique to eliminate artifacts in the raw data by scoring the corresponding raw-data for each epoch (i.e., the waveform section of EEG within 1 sec). The GRU machine learning process conducted in this study operates on a throughput of a minimum of 30,000 counts to a maximum of 35,000 counts of 32-channel raw-data quantification EEG measured over a duration of 1 min for each body posture type (class).
Location and name of 32 electroencephalogram channels: Frontal lobe 10, temporal lobe 8, parietal lobe 9, occipital lobe 3 channels, both ear electrodes 2 channels.
GRU data processing procedure
In this study, the application and investigation of GRU technology were conducted. The optimization of the GRU algorithm was prioritized to enhance the accuracy and reduce the loss value of the performance index. This was achieved through a comprehensive 4-step learning process. The raw-data preprocessing phase utilized the ScikitLearn Python module, while the deep learning network configuration was implemented using the Tensorflow platform. Following the collection of EEG experiment data, the GRU data processing procedure encompassed four key steps: (a) data preprocessing, (b) data correlation analysis, (c) GRU network learning calculation process, and (d) evaluation of performance indices.
Step 1. Data preprocessing process
The data preprocessing process, illustrated in Fig. 2, involves segmenting the EEG signal into frames. The upper part of Fig. 2 outlines the three steps of the data preprocessing procedure: random shuffling of data (Data Shuffle), standardization using a normal distribution (StandardScaler), and one-hot encoding (OneHotEncoder) prior to GRU machine learning. While the data is randomly shuffled, the labels for individual upright exercise postures are preserved through binarization. To leverage the deep learning recurrent neural network algorithm, the EEG signal is divided into frames, as depicted in the lower left of Fig. 2. For data optimization, a frame overlapping of 16 data points (equivalent to 1/8 sec) is set, resulting in a total of 9,100 frames for the entire dataset. Each frame of the GRU comprises 30 channels, with each channel consisting of 64 data points. The training and test data are randomly split in an 8:2 ratio for machine learning purposes. As an example, the bottom right of Fig. 2 showcases the 200th frame data (X train 200) from the training dataset.
Architecture of data preprocessing for electroencephalogram data driven deep learning. Data shuffle, standard-scaler, and one-hot encoding are included. GRU, gated recurrent unit.
Step 2. Data correlation analysis
The subsequent step in the data preprocessing process involves validating the performance of the GRU algorithm, as depicted in Figs. 3–5. Fig. 3 illustrates the EEG results obtained from six channels (FP1, F7, T7, CZ, TP8, P4) corresponding to five postural control types (class 0, 1, ..., 4). Unique frequencies and amplitudes are exhibited by the EEG waveforms of each posture type in channels FP1, T7, CZ, and P4 (Fig. 3A, C, D, F). Specifically, distinctive EEG patterns for type 3 (class 2) and type 5 (class 4) are observed in channels F7 and TP8 (Fig. 3B, E). Fig. 4 compares the appearance of EEG signals between type 5 (class 4) and type 3 (class 2). Differences in EEG signals between the two conditions, such as in the frontal lobe (FP1, F9), temporal lobe (FT1), and occipital lobe (O2), can be observed (Fig. 4). As one of the procedures for validating the GRU algorithm learning process, to examine the linear correlation between the two variables of the 32 EEG channels, Fig. 5 presents the Pearson correlation coefficient. It demonstrates a significant correlation between motor characteristics and posture types across 30 EEG channels for each difficulty level of the five posture control types. The correlation coefficients validate the effectiveness of the GRU algorithm in analyzing data correlations.
Comparative analysis of electroencephalogram time series signals during 5 types of upright postures. (A, B) Frontal lobes electroencephalogram signals (FP1, F7). (C, E) Temporal lobes electroencephalogram signals (T7, TP8). (D, F) Parietal lobe electroencephalogram signals (CZ, P4).
Comparative analysis of electroencephalogram cases: type 5 (class 4) and type 3 (class 2). (A) FP1: upper part depicts type 5 signal, lower part depicts type 3 signal. (B) F9: upper part depicts type 5 signal, lower part depicts type 3 signal. (C) FT1: upper part depicts type 5 signal, lower part depicts type 3 signal. (D) O2: upper part depicts type 5 signal, lower part depicts type 3 signal.
Pearson correlation coefficient values between electroencephalogram (EEG) 32 channel data. The closer the correlation coefficient is to +1, the brighter the color, and the closer to −1, the darker the color, where the color bar is ranged between ±1.
Step 3. GRU network structure and learning arithmetic process
Following data preprocessing, the subsequent step involves machine learning. Fig. 6 illustrates the structure and arithmetic process of the GRU network, an artificial intelligence technique used for machine learning. The left side of the GRU network structure represents the EEG data frame (EEG data with frames) categorized by class posture, which serves as the input layer (x1, x2, ... xn) for the learning process. The GRU layer, a component of the recurrent neural network, consists of a hidden layer where the posture recognition of human body movement is learned. To mitigate over-fitting, dropout is applied in the GRU layer. The hidden layer’s initial set weights are adjusted continuously through the back-propagation process. Subsequently, the information is propagated to the output layer (located at the right end of Fig. 6) via the fully connected layer. Finally, the performance index evaluation is conducted to assess the accuracy of exercise posture prediction.
Gated recurrent unit network structure. (A) Electroencephalogram (EEG) data with frames. (B) Input layers. (C) Gated recurrent unit (GRU) layer. (D) Fully connected layer. (E) Output layer for prediction of 5 classes.
Step 4. Evaluation of performance indicators
The evaluation of performance indexes in predicting artificial intelligence motion identification involves combining statistical processing and algorithms to uncover the relationship between brain waves and motion types and achieve model optimization. The optimization of the GRU model is determined by the accuracy of performance metrics and the cost function. Fig. 7 illustrates notable differences in weight prediction results when allowing or disallowing overlapping in the EEG data frame configuration. Fig. 7A and B represent the case without overlap, while Fig. 7C and D depict the accuracy and cost function (loss) results when overlap is permitted in the input layer’s frame setting. These performance index outcomes are learned in a manner that evaluates the overlapping weight of the most suitable model during the learning process, utilizing a setting condition where the optimization weight result of the validation data is assessed after each epoch (1-sec brain wave section). When allowing a 20% overlap, remarkable prediction accuracy is achieved in motion identification prediction, as demonstrated by the change in accuracy and cost function, which serve as performance indicators. Fig. 7A and C reveals that accuracy changes similarly between the machine learning (train accuracy) data and the validation accuracy data when overlap is allowed. Fig. 7D illustrates that the cost function (loss) steadily converges to approximately 0.1 around the fifth epoch when overlapping is permitted. Conversely, in Fig. 7B, where overlapping is not allowed, the cost function of the validation data exhibits significant fluctuations with each epoch, indicating the presence of overfitting.
Comparison of prediction results of gated recurrent unit optimization performance metrics. (A) Accuracy of allowed frame setting. (B) Loss values for the condition of overlapping allowed frame setting. (C) Accuracy of not-allowed frame setting. (D) Loss values for the condition of overlapping not-allowed frame setting.
RESULTS
The motion identification performance, enhanced by improving the performance index, was evaluated and compared using Fig. 8A, B. As depicted in Fig. 8A and B, the final accuracy performance exhibited significant improvement as indicated by the confusion matrix. Specifically, the model was able to predict the exercise type by accurately identifying the motion patterns and prediction results for the 5 posture types among the participants in the experiment. Notably, there were substantial enhancements in the accuracy performance of the 5 posture types. Sitting (type 1; class 0) saw an improvement of 5.49% (from 93.33% to 98.82%), while two-legged standing (type 2; class 1) experienced a boost of 7.32% (from 90.91% to 98.23%). Standing with eyes open (type 3; class 2) achieved the highest accuracy of 99.15% in both measured and predicted values (an increase of 2.52% from 96.63% to 99.15%). Among the five posture types, nonmain standing (type 4; class 3) demonstrated the greatest accuracy improvement, with a significant margin of 15.92% (from 78.75% to 94.67%). Additionally, standing with eyes closed (type 5; class 4) experienced a substantial improvement of 7.03% (from 89.02% to 96.05%). This research model achieved prediction accuracies ranging from a minimum of 94.67% to a maximum of 99.15%, surpassing the performance of the existing model. The probability of the exercise pattern of standing with eyes open (type 3; class 2) matching sitting posture (type 1; class 0), two-legged standing (type 2; class 1), and standing with eyes closed (type 5; class 4) was 0.00% each.
Comparison of confusion matrix performance of electroencephalogram data. (A) For the condition of overlapping allowed frame setting. (B) For the condition of overlapping not-allowed frame setting. X-axis is the predicted posture and y-axis is the actual posture, where class 0 (type 1), class 1 (type 2), class 2 (type 3), class 3 (type 4), class 4 (type 5).
Next, the results of discriminating between different postures within the same experiment participants were examined. The similarity between the dominant (right foot) and nondominant (left foot) main standing (type 3; class 2) and nondominant standing (type 4; class 3) was found to be 0.63%. When visual information was blocked, the agreement between standing with eyes open (type 4; class 3) and standing with eyes closed (type 5; class 4) was 0.00%, irrespective of the presence or absence of visual information for the same experimenter. This demonstrated a clear ability in posture recognition and movement identification. Summarizing the aforementioned results, it is evident that the posture identification accuracy (precision) of the AI-based GRU model, as reflected in the loss value and accuracy, surpassed that of the existing model. Therefore, the GRU algorithm model proposed in this study serves as an objective evaluation tool for postural control and motor identification based on EEG data collection.
DISCUSSION
The research question addressed in this study concludes that it is feasible to identify human postures and predict movements through machine learning and training of the GRU model, leveraging the strong correlation between brain activity and movement, particularly in terms of brain concentration. The application of artificial intelligence, specifically GRU, holds significant value as a practical research tool for overcoming the limitations of traditional quantitative EEG methods. It is anticipated to contribute to future research endeavors by mitigating subjective errors and limitations in visual interpretation during integrated EEG analysis. Previous studies (Choi et al., 2018) investigating the effects of exercise linked to EEG primarily focused on clinical pathologies such as dementia diagnosis and mental disorders, as well as aspects related to emotional recognition and learning outcomes. By integrating artificial intelligence techniques, comprehensive research on the complex human movement system becomes possible, leading to substantial advancements in the field of neuromechanics of the brain and neuromuscular system, which have long remained enigmatic.
The findings of this study indicate that during upright standing exercises, there exists interconnectivity and communication between the brain and exercise through exercise command signals. Frequency analysis using fast Fourier transform conversion during upright standing exercises has demonstrated a correspondence between brain-exercise frequency waveforms and frequency band signals at specific frequencies, such as brain action potential frequency (fEEG), plantar pressure center frequency (fCoP), and ankle joint torque frequency (ftorque) (Yoo, 2019). Notably, an increase in postural sway corresponds to higher vibration in the center of pressure and EEG signal band values, indicating proportional changes due to brain activation. Previous studies have supported this relationship by revealing a close correlation between electromyography signals and EEG signals in brain-musculoskeletal studies.
These research findings align with the writings of neuroscientist Damasio and Damasio (2022), who emphasizes that the brain maintains communication with the musculoskeletal system not only during exercise but also in everyday life, even without movement. This concept is reinforced by the emerging trend in brain-computer interfaces (BCIs) wherein researchers focus on developing technologies that enable the extraction of brain activity signals for motor control and somatosensory feedback, allowing for the restoration of lost motor functions through the development of decoding techniques (Hiremath et al., 2017). Consequently, it can be defined that exercise instructions originating from the central nervous system of the brain and exercise command signals from the musculoskeletal system, which governs extremity movements, work in unison to regulate body balance and postural stability (O’Reilly et al., 2022; Yoo, 2019).
The positive research outcomes achieved in motion identification prediction using AI-based brain wave analysis in this study hold immense potential as a robust research tool for advancing the field of biohealth BCIs in the era of the fourth industrial revolution (Baek, 2016; Chen et al., 2022; Eun et al., 2021; Hamet and Tremblay, 2017). In a similar application study, Liao et al. (2020) achieved a remarkable accuracy of 98.88% in estimating cybersickness caused by sensory-cognitive system mismatch during virtual reality experiences by utilizing brain wave data. Furthermore, Jeon et al. (2019) obtained a performance index with a sleep stage discrimination accuracy of up to 92% through the automatic classification of brain waves.
This study aimed to predict movements by developing a deep learning GRU model that uncovers the strong correlation between brain activity and movement using EEG data. Through optimization of the GRU model, the performance index was improved by 15.92% compared to the accuracy of the existing model. Moreover, the motion prediction recognition rate ranged from a minimum of 94.67% to 99.15% depending on the specific posture type. These research findings are deemed significant contributions to the field of brain science, as they leverage the innovative GRU algorithm and machine learning techniques applied to brain wave information. This study has the potential to establish a new academic paradigm for exercise rehabilitation research.
Notes
CONFLICT OF INTEREST
No potential conflict of interest relevant to this article was reported.
ACKNOWLEDGMENTS
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2019S1A5A2A01047204).