Analysis of intrarater and interrater reliability of the interactive line protocol for evaluating neuromuscular function in patients with knee injury and observation of psychological state changes
Article information
Abstract
The isokinetic interactive line protocol (ILP) appears to enhance neuromuscular function and may reduce posttraumatic stress and anxiety. However, the reliability of the protocol has not been investigated, nor has it been analyzed in relation to psychological states. A total of 165 participants took part in this study, with ages ranging from 21 to 24 years. The gender distribution was 68.4% male and 31.6% female. There was no significant difference in injury duration between males and females (z=−0.922, P=0.357). The intrarater reliability of the ILP, as indicated by the intraclass correlation coefficient, was high for both concentric and eccentric contractions of the quadriceps (ranging from 0.891 to 0.980). Likewise, the interrater reliability for these contractions was also high (range, 0.873–0.943). Additionally, during the ILP, there was a significant reduction in posttraumatic stress (t=9.795, P<0.001) and anxiety (t=5.965, P<0.001) related to knee reinjury after 2 weeks. Ultimately, the isokinetic game ILP demonstrates high intra- and interrater reliabilities and can be concluded to effectively reduce posttraumatic stress and anxiety when incorporated into retesting.
INTRODUCTION
Isokinetic dynamometry is regarded as a reliable tool for measuring muscle strength and is frequently used as a benchmark for other strength evaluations (Stark et al., 2011). An isokinetic dynamometer enables the evaluation of muscle function by providing accommodating resistance at a consistent angular velocity (Habets et al., 2018). In clinical practice, isokinetic dynamometers are often used to assess and monitor progress during rehabilitation, and various studies have been conducted to ensure the reliability of test results obtained from the same dynamometer (Maffiuletti et al., 2007). For these tests to be clinically meaningful, the assessment procedure must be reliable and sensitive enough to determine whether the findings indicate an injury and to evaluate the outcomes of therapeutic interventions (Sole et al., 2007). In other words, the measurements should be defined in a way that helps determine whether the observed changes fall within the range of assessment error or represent actual changes.
Recently, with the advent of the gaming industry era, rehabilitation exercises and assessment methods utilizing game modes have been introduced (Jee, 2017). Although isokinetic rehabilitation equipment includes software developed and installed as part of the gaming industry, it is not yet widely used, and there has been no research on its reliability (Javaid et al., 2020). In other words, while game-based exercises or assessments might engage patients by incorporating physical, cognitive, or psychological elements, potentially providing greater motivation compared to traditional methods (Jee, 2017), there is a lack of research on their reliability (Eng et al., 2002). It is also believed that using gamified software could help reduce the anxiety associated with rehabilitation exercises in a new environment and the posttraumatic stress that can easily occur after an injury. In other words, anxiety about reinjury may increase when moving the injured area for posttraumatic stress disorder or physical rehabilitation. However, applying exercise rehabilitation in the form of a game could potentially reduce the aforementioned psychological distress, although no research on this topic.
Isokinetic testing is widely recognized as an excellent quantitative method for measuring muscle torque and is commonly used in the field of rehabilitation exercise. However, there is still considerable debate regarding the results of this test. Some studies have reported a low correlation between isokinetic tests using dynamometers and muscle strength assessments for specific muscles (Noreau and Vachon, 1998). Additionally, when evaluating the reliability of isokinetic testing for assessing knee muscle spasticity in certain patients, it has been found that reliability is high at fast speeds but low at slow speeds (Pierce et al., 2006). Specifically, the interactive line protocol (ILP) integrated within the isokinetic equipment, which appears promising for assessing and training neuromuscular function, has yet to have its reliability established. Moreover, no studies have examined the reliability of this assessment method in patients with knee disorders. Therefore, the purpose of this study was to investigate the interrater and intrarater reliability of the ILP, a tool developed to improve neuromuscular function, and to observe how this tool affects posttraumatic stress and anxiety levels.
MATERIALS AND METHODS
Participants
The study sample included participants aged 21 to 24 years, focusing on young adults with a history of knee injuries. The duration related to the knee injuries of the participants in this study ranged from a minimum of 1.6 years to a maximum of 3.3 years, and all reported that the injuries were caused by sports activities. The types of sports were soccer and basketball. Individuals with musculoskeletal issues in other parts of the body or specific medical conditions were excluded. Exclusion criteria also included participants who had sustained lower and/or upper extremity injuries limiting their physical activity in the past 6 months, those who had undergone surgery on the lower and/or upper extremities within the previous 12 months, those with a diagnosis that could interfere with proper test execution, or those taking medications that could affect their ability to perform the ILP. The sample size was calculated using the GPower program for a basic correlation research design with a bivariate normal model, using parameters of two tails, a correlation ρ H1 of 0.3, an α error probability of 0.05, a power (1-β error probability) of 0.95, and a correlation ρ H0 of 0 (Faul et al., 2009). The minimum required sample size was 138 participants. To account for an estimated 20% dropout rate (Sedgwick, 2013), 165 participants were recruited.
Experimental design and procedure
This study aimed to assess the reliability of the ILP, a gamified exercise rehabilitation protocol for patients with knee injuries, by evaluating both intrarater and interrater reliability. The type of gamified exercise used was the ILP embedded within the isokinetic machine (HUMAC/NORM Testing and Rehabilitation System, CSMi, Stoughton, MA, USA). The test and retest were conducted with a 2-week interval, and the psychological state of the knee injury patients, including posttraumatic stress and test-related anxiety, was measured before and after the tests to observe any changes. Data for this study were collected from March 1 to March 15, 2024. The study was approved by the Hanseo University Human Institutional Review Board (HS23-11-23-2).
Body composition and controlled variables measures
Body composition was evaluated through bioelectrical impedance analysis using a body composition analyzer (Inbody 770, Biospace, Seoul, Korea) after manually measuring participants’ heights. Daily dietary intake was recorded with CAN-Pro 5.0 (Korean Nutrition Society, Seoul, Korea, 2023), while physical activity levels were tracked using the International Physical Activity Questionnaire (Booth, 2000). Both sets of data were recorded daily on a computer and averaged at the end of each week (Heo et al., 2023).
ILP assessment and procedure
The software installed on the computer was calibrated prior to the start of the study according to the operating manual. In this study, the program used was a gamified ILP, where participants aimed to score high by keeping two balls close to a target line. Specifically, as the two designated balls move up the line, the quadriceps engage in concentric contractions, while moving the balls down the line requires eccentric contraction of the quadriceps. The intensity for concentric contractions was set at 10 Nm, and for eccentric contractions, it was set at 13 Nm, with each session lasting 1 min according to the operating manual. The position of the balls represented torque generation, with the x-axis displaying the percentage of maximal voluntary contraction, which was then converted into a score. If the balls remained green, it indicated high accuracy, whereas turning red signified that the target was missed. Throughout the entire duration of the ILP, all participants received visual feedback from the system monitor. However, the researchers only instructed the participants on the methods and did not provide any additional encouragement. For intrarater reliability, one examiner repeated the measurements after 2 weeks. Meanwhile, to assess interrater reliability, two examiners participated in the measurements. The order of measurement was determined by having the study participants randomly draw a ball from a bag containing numbered balls that corresponded to the examiners, indicating the order in which the two examiners would perform the measurements. In other words, the two examiners conducted their measurements in a randomly assigned order to evaluate interrater reliability.
Posttraumatic stress and anxiety measures
In addition to assessing the reliability of the ILP, this study aimed to determine the level of stress or anxiety that patients with knee injuries may experience during the examination. First, this study used the abbreviated version of the posttraumatic stress disorder (PTSD) Checklist for DSM-5 (PCL-5) to assess PTSD in patients with knee injuries (Zuromski et al., 2019). The questionnaire includes 4 items. Each item is rated on a Likert scale, with ‘0’ indicating no PTSD symptoms and ‘4’ indicating severe symptoms (Zuromski et al., 2019). The reliability coefficient, measured by Cronbach α, was 0.702, demonstrating satisfactory internal consistency. Second, a modified version of the visual analog scale was used to measure anxiety levels (Williams et al., 2010). Participants were asked to indicate their self-perceived state in response to the researcher’s questions. Specifically, they were instructed to rate their “Concerns about potential anxiety before participating in an isokinetic interactive game” on a 10-cm bipolar scale. Those who experienced no anxiety prior to the interactive line program marked ‘V’ near ‘0,’ while those with extreme anxiety marked closer to ‘10.’ The questionnaire’s reliability was assessed using Cronbach α, with the anxiety scale demonstrating a coefficient of 0.629, indicating good internal consistency.
Statistical analysis
All data were analyzed using IBM SPSS Statistics ver. 23.0 (IBM Co., Armonk, NY, USA). Additionally, GraphPad Prism 10.3 (GraphPad Software, Boston, MA, USA) was utilized to create graphical representations. Data are presented as mean±standard deviation. The normality of demographic, anthropometric, and clinical measures was assessed with the Kolmogorov–Smirnov test. Categorical variables, expressed as n (%), were compared using appropriate statistical tests such as chi-square or Fisher exact test. For normally distributed data, variable comparisons were made using a paired t-test, while the Wilcoxon signed-rank test was applied to nonnormally distributed data. The correlation between the initial and follow-up assessments was examined using Pearson correlation coefficient. In this study, intraclass correlation coefficient (ICC) values below 0.40 are regarded as poor, those between 0.40 and 0.59 as fair, values between 0.60 and 0.74 as good, and values between 0.75 and 1.00 as excellent (Weir, 2005). Detailed data analysis involved calculating the change over each period as delta (Δ)%. Statistical significance was defined as P≤0.05.
RESULTS
Demographic and physical characteristics
As shown in Table 1, the participants’ ages ranged from 21 to 24 years, with 23 years being the most common age at 37.6%. The sex distribution was 68.4% male and 31.6% female. There was no significant difference in the duration of injury between males and females (z=−0.922, P=0.357). Additionally, there were no significant differences in diet and physical activity levels measured before (z=−1.098; P=0.272) and after (z=−0.728; P=0.466) conducting the ILP.
Demographic and physical characteristics
Intrarater reliability of an ILP
To assess the intrarater reliability of the ILP, the ICC was calculated. The reliability coefficients measured during concentric and eccentric contractions for the quadriceps are presented in Table 2. After performing ILP, the accuracy scores for the quadriceps’ concentric contraction were found to have high reliability. Similarly, the accuracy scores for the quadriceps’ eccentric contraction also demonstrated high reliability after performing ILP. Meanwhile, as shown in Fig. 1, the intrarater test-retest correlations for concentric contraction score (Fig. 1A) and eccentric contraction score (Fig. 1B) of the quadriceps were found to be high.
Intrarater reliability of the quadriceps contraction on executing an ILP
Intrarater test-retest correlation for concentric (A) and eccentric (B) contraction scores of the quadriceps.
Interrater reliability of an ILP
To evaluate the interrater reliability of the ILP, the ICC was calculated. Table 3 presents the reliability coefficients measured during concentric and eccentric contractions of the quadriceps. Following the ILP, the accuracy scores for the quadriceps during concentric contraction exhibited high reliability. Likewise, the accuracy scores for the quadriceps during eccentric contraction also showed high reliability after ILP. As shown in Fig. 2, the interrater test-retest correlations for concentric contraction score (Fig. 2A) and eccentric contraction score (Fig. 2B) of the quadriceps were found to be high.
Interrater reliability of the quadriceps contraction on executing an ILP
Interrater test-retest correlation for concentric (A) and eccentric (B) contraction scores of the quadriceps.
Changes of posttraumatic stress and anxiety levels
Among the four questions related to PTSD, the score for the first question decreased from 2.29±0.85 before the test to 1.88± 0.65 afterward, reflecting a 10.76% reduction. This change was statistically significant (t=6.103, P<0.001). For question 2, the score decreased from 2.13±0.75 before the test to 1.92±0.57 two weeks later, showing a 2.73% reduction, which was statistically significant (t=3.515; P=0.001). For question 3, the score reduced from 2.24±0.71 before the test to 1.96±0.57 two weeks later, indicating a 7.93% decrease, and this change was statistically significant (t=4.969; P<0.001). For question 4, the score fell from 2.45±0.84 before the test to 2.04±0.76 two weeks later, reflecting an 11.11% reduction, with a statistically significant change (t=6.089; P<0.001). The sum of the four PTSD questions decreased from 9.11±1.82 before the test to 7.81±1.38 two weeks later, representing a 12.29% reduction, which was statistically significant (t=9.795; P<0.001) (Fig. 3A). Similarly, the anxiety level decreased from 4.67±1.64 before the test to 3.93±1.41 two weeks later, reflecting a 5.03% reduction, with a statistically significant change (t=5.965; P<0.001) (Fig. 3B).
Changes in posttraumatic stress disorder (PTSD; A) and anxiety (B) scores after 2 weeks. ***P<0.001.
DISCUSSION
This study established that the isokinetic ILP exhibits high intrarater and interrater reliability. Additionally, it found that patients with knee injuries experience a reduction in previous posttraumatic stress and anxiety related to the protocol when they undergo ILP again.
To evaluate the reproducibility of clinical tests, test-retest reliability is most used. This approach allows for the calculation of the ICC and standard error, which are then utilized in the analysis (Mehta et al., 2018). To our knowledge, the ICC values we generally recognize range from 0 to 1, with values above 0.75 indicating good reliability, while values below 0.75 indicate reliability ranging from poor to moderate. However, an ICC between 1 and 0.90 is interpreted as ‘very high’ reliability, 0.89 to 0.70 as ‘high,’ 0.69 to 0.50 as ‘moderate,’ 0.49 to 0.26 as ‘low,’ and a value below 0.25 as ‘very low’ according to Tsekoura et al. (2021). Similarly, Weir (2005) considers an ICC value below 0.40 as poor, values between 0.40 and 0.59 as fair, values between 0.60 and 0.74 as good, and values between 0.75 and 1.00 as excellent. The reliability of a tool can vary depending on whether it is assessed through intrarater or interrater methods. Taking this into account, this study examined the reliability of the ILP using both approaches. First, to evaluate the intrarater reliability of the ILP, the ICC was calculated. As shown in the results of this study, the ICC for the accuracy scores of concentric contractions of the affected quadriceps during ILP, compared with the scores obtained from the same limb after 2 weeks, was 0.902, indicating high reliability. Similarly, the accuracy scores for eccentric contractions of the quadriceps within the same evaluator also showed a high ICC of 0.891 between the first and second attempts of performing the ILP, further demonstrating high reliability. On the other hand, the ICC for the accuracy scores of concentric and eccentric contractions of the quadriceps on the unaffected side were 0.951 and 0.980, respectively, showing slightly higher reliability compared to the affected side. Secondly, in the assessment of interrater reliability for the ILP, the ICC for the accuracy scores of concentric contractions of the affected quadriceps, compared with the results after two weeks, was 0.873, and the ICC for the accuracy scores of eccentric contractions was 0.895. On the other hand, the ICC for the accuracy scores of concentric and eccentric contractions of the quadriceps on the unaffected side were 0.943 and 0.918, respectively, showing slightly higher reliability compared to the affected side. These results exceed the high ICC range suggested by various scholars, indicating that the ILP may be highly effective when applied to patients with knee injuries. Furthermore, the strong correlations observed in this study between tests and retests, both within and between evaluators, reinforce the significance of these findings.
In fact, posttraumatic stress refers to a condition where individuals experience fear and persistent distress as a result of repeatedly re-experiencing traumatic events, such as severe incidents (Koenen et al., 2017). This condition can prevent individuals with posttraumatic stress from performing everyday social functions, as they make extensive efforts to cope with and escape their distress. Posttraumatic stress related to knee joint injuries refers to the psychological and emotional distress experienced by individuals who have suffered a traumatic event or injury affecting the knee. This distress has been reported to lead to avoidance of various social activities (Starr et al., 2004; Zuromski et al., 2019). Individuals with knee joint injuries may experience symptoms of posttraumatic stress such as flashbacks, nightmares, avoidance behaviors, negative changes in mood and cognition, and hyperarousal (Van Der Kolk, 2001). Similar to postinjury stress symptoms, patients often report anxiety when performing certain movements or activities (Vincent et al., 2015). This anxiety can be heightened even when attempting to rehabilitate the injured area in a new environment (Daley et al., 2021), underscoring the need for immediate and long-term treatment methods. In this study, the level of posttraumatic stress was measured using the PCL-5 short form among patients with a history of knee injury before applying the ILP. The results were as follows: PTSD#1 had a score of 2.29±0.85, PTSD#2 had a score of 2.13±0.75, PTSD#3 had a score of 2.24±0.71, and PTSD#4 had a score of 2.45±0.84, with a total score of 9.11± 1.82, indicating a moderate level of posttraumatic stress. Additionally, the anxiety level before applying the ILP was 4.67±1.64, suggesting that patients experienced a certain degree of anxiety about rehabilitation exercises in a new environment. On the other hand, during the test-retest process, measurements of posttraumatic stress and anxiety levels using the same questionnaires showed that PTSD decreased by as much as −11.11% and as little as −2.37%. Anxiety also decreased by −5.03%. These results indicate that the ILP provided patients with a sense of familiarity and stability, contributing to reduced posttraumatic stress and anxiety. Indeed, posttraumatic stress and anxiety are common among patients and are associated with significant impairment (Koenen et al., 2017), which often leads to lower treatment success rates (Thornicroft, 2018). This suggests that traditional rehabilitation methods may have limitations and highlights the need for programs or protocols that are both reliable and familiar to patients. The ILP used in this study, which is delivered in a gamified format, was perceived as somewhat more comfortable for patients with knee injuries. Additionally, the scientific structure of the ILP, integrated within an isokinetic device, likely enhanced its credibility and effectiveness. However, despite these positive results, there are limitations. This study focused exclusively on patients with knee injuries, even though there are many different musculoskeletal disorders. Additionally, the study was limited to younger individuals, and while various protocols are available within isokinetic devices, only the ILP was utilized. Future research should address these limitations by including a broader range of participants and integrating different protocols.
To our knowledge, this study confirmed that the isokinetic ILP for the patients with injured knee demonstrates high intra- and interrater reliability. It also found that when patients with knee injuries undergo ILP again, their previous posttraumatic stress and anxiety related to the protocol are reduced.
Notes
CONFLICT OF INTEREST
No potential conflict of interest relevant to this article was reported.
ACKNOWLEDGMENTS
This research was supported by the 2024 Hanseo University Research Grant.