The Impact of “Homework” Practice on Results for Digital Therapeutics
This is article part of a series of articles exploring methods for improving the results for the online program. This article investigates whether completion of home assignments is associated with greater improvement on well researched self-report outcome questionnaires for adults utilizing an online self-directed cognitive behavioral therapy program. Results suggest that completion of homework assignments was associated with greater improvement even after controlling for other variables also known to be associated with better results. The implications for the widening use of online CBT programs are discussed.
The value of internet-based adaptations of psychotherapeutic techniques has been well documented, especially since numerous products, now called digital therapeutics, have been tested and sold over the past decade. The authors have studied one such product, Learn to Live (learntolive.com), to determine what activities might enhance those clinical results.
The Learn to Live program routinely asks users to complete validated outcome questionnaires at every lesson, including commonly used tools such as the PHQ-9 and GAD-7. Questionnaires are selected based on the focus of each clinical program or module. Brown et al. (2020) used this rich dataset to compare results for the program to a reference sample of patients engaged in outpatient therapy.
The earliest study (Brown & Jones, 2020) explores how support impacts results by examining participation in a voluntary coaching program provided free of charge to all users. Those taking advantage of coaching displayed significantly greater improvement in earlier lessons, and they had a higher retention rate after each lesson. These results support the assertion that coaching significantly enhances the effectiveness of internet-based CBT programs.
The second support option investigated, automatically generated “mindfulness texts,” are messages sent to users to inspire and motivate them (Brown & Jones, 2022a). It is noteworthy that people receiving support through these low-cost mindfulness text messages improved significantly. This finding has important implications for the question of how to provide cost-effective services.
The third form of support studied (Brown & Jones, 2022b) is the program’s “Teammates” feature which allows users to list up to two friends or family members to be contacted for support during completion of the program. Those users selecting Teammates showed a level of improvement comparable to people engaged in outpatient therapy. The findings suggested Teammates has a unique or distinctive effect in that there is an additive effect of combining it with coaching.
The three previous articles established that various types of support can enhance clinical outcomes, and the current study focuses on how the practicing of new psychological skills, traditionally called homework by proponents of cognitive-behavioral therapy, can improve results. This is a preliminary study, made possible by program enhancements in 2021 to better track such user behavior on the platform.
These recent changes afford a shorter time interval to generate results. Accordingly, the available sample size is smaller than earlier cohorts studied, but the results are sufficiently encouraging to warrant sharing them now with researchers and designers of digital therapeutics.
CBT and Homework
The value of homework has long been a key tenet of CBT. However, the number of studies investigating the impact of homework completion on treatment outcomes is limited, and most studies are based on small sample sizes. While the impact of practicing new skills with homework exercises seems positive in these few studies, the impact of practice exercises with digital CBT programs is untested.
Kazantzis et al. (2000) published a meta-analysis of 27 in-person CBT studies, with 1,702 subjects, which breaks down to an average of 63 subjects per study. This meta-analysis found overall support for the premise that completion of homework is associated with greater improvement in therapy.
Carol et al. (2005) found a positive association between homework completion and outcome for 22 patients receiving treatment for cocaine addiction. Neimeyer et al. (2008) evaluated results for a ten-week group therapy CBT program (sample=46) and found that homework completion was associated with better outcomes.
Rees et al. (2008) evaluated results from a ten-week CBT group therapy program (sample=94), finding that the strongest association was between the number of homework assignments completed and the treatment outcome after ten weeks. The relevance of the homework to the patient’s immediate problem was less predictive, leading the authors to conclude: “The results were strongest for the amount of homework completed, suggesting that clinicians should encourage patients to complete homework even if the homework content is not entirely accurate.”
Helbig-Lang et al. (2015) report on a study of 292 patients with panic disorder and agoraphobia and treated with standardized CBT. They likewise evaluated both the quantity and relevance of the homework. While both were associated with better outcomes, the authors found that the relevance or quality of the homework was a better predictor.
Haller & Watzke (2021) conducted an analysis of homework engagement for 22 subjects receiving CBT via phone sessions. Sessions were recorded and coded as to whether the subject engaged in homework activity. This study likewise found a positive association between homework completed and outcome.
In summary, limited as it is, the research supports the premise that engaging users of digital CBT platform in completion of suggested homework assignments may contribute outcomes.
Description of sample
The sample analyzed for this study is a cohort of users who registered to use the platform from March 2021 through May 2022. The dataset contained information on the number of homework assignments each person completed. This information was not available in previous studies of the Learn to Live platform.
The sample includes 670 users completing at least two lessons of any clinical module. This requirement is necessary to calculate change scores and effect size. Users are classified as being in the clinical range or the normal range at intake. This calculation depends on the clinical cutoff scores previously established, and they are based on patients in outpatient psychotherapy.
In the sample under study, 78% of the users (n=525) were classified as being in the clinical range at intake. Of these, 108 (21%) completed at least one homework assignment. While this is not a large sample, it is comparable to sample sizes reported in other studies.
The analysis is restricted to those in the clinical range since this is consistent with published psychotherapy research and the methodology established by the ACORN collaboration. This permits the use of effect size calculations and meta-analyses that examine people with enough distress at intake to measure change reliably.
The mean effect size for this cohort was .75. While this is a relatively large effect size, comparable to that observed routine outpatient psychotherapy, this amount of improvement is slightly less than the mean effect size of .83 calculated for previous cohorts using the platform. This is a relatively small sample, and so these results may not represent a meaningful difference. Future cross validation will provide a better estimate of overall effect sizes.
The mean effect size for users completing at least one homework assignment (.86) was greater than that for people not doing any practice exercises (.72). An effect size difference of .16 is large and clinically meaningful if these results hold up on cross validation with larger sample size.
Positive correlations are found between the completion of lessons, homework exercises, and effect size. Specifically, the correlation between lessons completed and effect size is .32 (p<.001), while the correlation between the number of homework exercises and the lessons completed is .25 (p<.001). This suggests the possibility that homework effects might simply be due to additional lessons completed.
However, further analysis suggests a different understanding. Homework was studied in connection with the previously noted benefits of getting support while using the platform. The completion of at least one homework exercise was included in a multivariate model including each form of support (coaching, texting, and teammates). The model uses the last lesson completed to predict effect size.
The results suggest an additive contribution to effect size by completing homework exercises. As expected, the number of lessons completed is the strongest predictor of effect size (p<.001). However, the second strongest predictor is completion of at least one homework assignment (p<.05) after controlling for the various forms of support. These findings provide support for the premise that completing homework exercises makes a positive contribution to outcomes for users of the LearnToLive platform.
The results are consistent with prior research on the positive impact of homework assignments in CBT and extends this to users of an online digital platform to deliver CBT based lessons. Importantly for the present digital application, the finding offers yet another pathway to improving outcomes by exploring methods to shape users’ choices to selection options within the application that have been shown to improve user retention and outcomes. These include various forms of social support, and now encouraging user to complete homework can be added to the list of options for user to achieve better outcomes.
Cite This Article
Brown, J., & Jones, E. (2022, October). The impact of “homework” practice on results for digital therapeutics. [Web article]. Retrieved from http://www.societyforpsychotherapy.org/the-impact-of-homework-practice-on-results-for-digital-therapeutics
Brown, J. S., Jones, E., & Cazauvieilh, C. (2020). Effectiveness for online cognitive behavioral therapy versus outpatient treatment: A session by session analysis. http://www.societyforpsychotherapy.org/effectiveness-for-online-cognitive-behavioral-therapy-versus-outpatient-treatment
Brown, J.S., & Jones, E. (2020). Impact of coaching on rates of utilization and clinical change for digital self-care modules based on cognitive behavioral therapy. http://www.societyforpsychotherapy.org/impact-of-coaching-on-rates-of-utilization-and-clinical-change-for-digital-self-care-modules-based-on-cognitive-behavioral-therapy
Brown, J., & Jones, E. (2022a). Improving results for digital therapeutics. http://www.societyforpsychotherapy.org/improving-results-for-digital-therapeutics
Brown, J., & Jones, E. (2022b). Improving results for digital therapeutics with social support. http://www.societyforpsychotherapy.org/improving-results-for-digital-therapeutics-with-social-support
Carroll, K. M., Nich, C., & Ball, S. A. (2005). Practice Makes Progress? Homework Assignments and Outcome in Treatment of Cocaine Dependence. Journal of Consulting and Clinical Psychology, 73(4), 749–755. https://doi.org/10.1037/0022-006X.73.4.749
Haller, E. & Watzke, B. (2021). The Role of Homework Engagement, Homework-Related Therapist Behaviors, and Their Association with Depressive Symptoms in Telephone-Based CBT for Depression. Cogn Ther Res 45, 224–235. https://doi.org/10.1007/s10608-020-10136-x
Helbig-Lang, S.,Lang, T., Gloster, A.T., Fehm, L., Gerlach, A.L., Ströhle, A., Deckert,J., Kircher, T., Hamm, A.O., Alpers, G.W., Arolt, V., & Wittchen, H.U. (2013). Specificity of Homework Compliance Effects on Treatment Outcome in CBT: Evidence from a Controlled Trial on Panic Disorder and Agoraphobia, Journal of Clinical Psychology, 69(6), 616-629. https://doi.org/10.1002/jclp.21975
Kazantzis, N., Deane, F. P., & Ronan, K. R. (2000). Homework assignments in cognitive and behavioral therapy: A meta-analysis. Clinical Psychology: Science and Practice, 7(2), 189–202. https://doi.org/10.1093/clipsy.7.2.189
Neimeyer, R.A., Kazantzis, N., Kassler, D.M., Baker, K.D. & Fletcher, R. (2008) Group Cognitive Behavioural Therapy for Depression Outcomes Predicted by Willingness to Engage in Homework, Compliance with Homework, and Cognitive Restructuring Skill Acquisition, Cognitive Behaviour Therapy, 37(4), 199-215, DOI: 10.1080/16506070801981240
Rees, C.S. , McEnvoy, P. & Natan, P.R. (2005) Relationship between homework completion and outcome in cognitive behaviour therapy, Cognitive Behaviour Therapy, 34(4), 242-247, DOI: 10.1080/16506070510011548