Predicting the Therapist Effect
Study rationale and what we have learned without even looking at the data
Clinical Impact Statement: This article describes a study funded by Division 29 that seeks to better understand what makes therapists effective. The importance of this line of inquiry is addressed, along with the rationale for a fully remote study design. In addition, early lessons learned through the planning and data collection process are discussed.
At the time of this writing, an estimated one-third of the world’s population is in “lockdown” due to a novel coronavirus (Kaplan et al., 2020). In addition to the loss of life and physical health consequences, the economic and psychological impact of the virus and these containment measures has already been massive and may well be felt for years to come (Brooks et al., in press; Jones et al., 2020).
It would seem tone-deaf to not acknowledge the widespread human suffering at this moment in history, whether in the form of illness, unemployment, or racism (Bui & Wolfers, 2020; Tavernise & Oppel, 2020). This period represents perhaps the most rapid change in human social behavior that has ever occurred (D. Coogan, personal communication, March 27, 2020) and may initiate a mental health crisis (World Health Organization, 2020).
The current and impending psychological pain highlights the hugely important role that mental health providers play. Psychotherapy and psychotherapists will be essential for the collective healing from this pandemic. Thankfully, there is longstanding evidence that psychotherapy is generally effective (Smith & Glass, 1977; Wampold & Imel, 2015). However, there is room for improvement, and unfortunately, many do not benefit (e.g., due to premature termination; Swift & Greenberg, 2012). The questions then become: How can we make psychotherapy more effective? And, how can we best support our patients in these times and in all times?
Most readers of the Psychotherapy Bulletin are likely already convinced that the therapist matters. But this is not just our bias; there is strong scientific evidence that this is the case. Two recent meta-analyses have sought to quantify the degree of variance in patient outcomes attributable to the therapist (i.e., the therapist effect; Baldwin & Imel, 2013; Johns et al., 2019). Both studies found that therapists account for approximately 5% of the variance in patient outcomes. Although 5% may seem small, bear in mind that this effect size is not wildly different from that linking therapeutic alliance and outcome (i.e., 7.7%; Flückiger, Del Re et al., 2018) and is roughly five times greater than that attributable to differences between psychotherapies (i.e., 1%; Wampold & Imel, 2015; Wampold et al., 1997).
Research estimating the size of the therapist effect has been essential for advancing our scientific understanding of psychotherapy. Yet simply saying that therapists account for variance in patient outcomes is akin to saying that diet accounts for variance in health. One still needs to know what to eat or not eat. Crucial next steps for work in this area include determining precisely what therapist characteristics predict this variation and, ultimately, how these therapist-level factors can be selected for and augmented through training.
Although consensus has not been reached on the characteristics of highly effective therapists (Wampold & Imel, 2015), psychotherapy researchers have been investigating the relationship between therapist characteristics and patient outcomes for decades. In the past 40 years, many therapist-level variables have been explored, from therapists’ personal therapy (Garfield & Bergin, 1971) and performance on cognitive tasks (Mintz et al., 1976) to interpersonal skills (Anderson et al., 2009) and professional self-doubt (Nissen-Lie et al., 2013). This literature has been systematically reviewed in two recent studies. Lingiardi and colleagues (2017) examined the association between therapists’ subjective and therapy-nonspecific characteristics with patient outcomes in psychodynamic psychotherapy. Across 30 studies, they found the most consistent evidence for therapists’ interpersonal skills and interpersonal functioning. Patients of therapists who were more affiliative and more interpersonally skilled tended to have better outcomes. Heinonen and Nissen-Lie (in press) reviewed 31 studies examining therapists’ professional and personal characteristics, finding performance-based measures of professional interpersonal skills to be one of the most consistent predictors of patient outcomes. Self-rated social skills, in contrast, did not predict outcomes.
The studies reviewed by Lingiardi and colleagues (2017) and Heinonen and Nissen-Lie (in press) provide an important foundation for clarifying what may account for the therapist effect. However, this literature has substantial shortcomings. Both systematic reviews remarked on methodological limitations, including the use of small samples (of therapists and patients per therapist; Heinonen & Nissen-Lie, in press) and the wide variability in predictors examined (Lingiardi et al., 2017), with an associated small number of direct replications. In addition, with some notable exceptions (Anderson et al., 2009; Schöttke et al., 2017), many studies used self-report measures with known biases (e.g., social desirability). Further, variation in analytic methods and inconsistent use of standardized effect sizes makes it difficult to determine the consistency or magnitude of the association between therapist factors and patient outcomes. At the end of the day, we, as a field, remain without clear empirical guidance specifying precisely which therapist characteristics are most important. The current study is designed to address some of these limitations in the hopes of clarifying measurable therapist characteristics that relate to treatment outcome.
In partnership with Drs. Jesse Owen and Mark Kopta at CelestHealth, recruitment has occurred through clinics using the Behavioral Health Measure (BHM; Kopta et al., 2015). In addition, we have partnered with Drs. Robbie Babins-Wagner and Amy Bender at the Calgary Counselling Centre. All staff therapists and trainees at partnering sites are eligible to participate in the study. Data collected from therapists will later be matched with patient outcome data routinely collected at participating sites. We aim to recruit ≥ 100 therapists whose responses will then be associated with data from their patients (anticipated patient n ≥ 1000). This target sample size of therapists would provide adequate power (beta = .80) based on a three-level power analysis that assumes a therapist intraclass correlation of .05 (Baldwin & Imel, 2013), 10 patients per therapist, 8 sessions per patient (Goldberg et al., 2018), and a small to moderate effect size (r ≥ .11, d ≥ 0.22).
Therapist recruitment is occurring through emails sent by clinic directors at partnering clinics. Study staff direct participants to an online survey that includes approximately two hours of self-report questionnaires and behavioral tasks. Participants are compensated $200 for their time.
In planning this study, we aimed to collect a wide variety of therapist variables that have been previously shown to predict patient outcomes or are theorized to be linked with patient outcomes. Recent reviews of the literature on therapist characteristics and patient outcomes highlight therapists’ interpersonal skills as one of the more consistent predictors. Therefore, a primary measure we are collecting is the Facilitative Interpersonal Skills task (FIS; Anderson et al., 2009). The FIS involves collecting participants’ verbal responses to a series of video-based vignettes depicting challenging moments in therapy. Responses are then coded along several domains (e.g., empathy, emotional expression). We have also included self-report and behavioral measures designed to assess component parts of the FIS (e.g., verbal fluency, empathy).
A second primary task we are using is the Multicultural Orientation task (MCO; Owen et al., 2018). Similar to the FIS, the MCO involves viewing a series of video-based vignettes to which participants are asked to respond. These vignettes depict multiculturally salient moments in therapy. Just as for the FIS, we have included other self-report and behavioral measures to more richly assess participants’ multicultural orientation.
Data Analytic Plan
Data will be analyzed using multilevel models that account for the nesting of patients within therapists. In these models, therapist characteristics will be entered as predictors of patient outcomes (e.g., change in outcome measures, early termination). We plan to examine potential interactions between patient demographics and therapist characteristics as predictors of outcomes through random slope models (e.g., Thompson et al., 2018). Further, we plan to use these data to continue the development of machine learning algorithms to automate the scoring of the FIS (Goldberg et al., in press). In keeping with recent efforts to increase transparency and replicability in psychology and science broadly, we plan to pre-register primary study hypotheses before data analysis (Open Science Collaboration, 2012).
We are still actively in the process of data collection for this study. As we have not yet pre-registered our study hypotheses, we have not begun analyzing our data. Nevertheless, we have learned a lot through the design and initiation of this study.
The Importance of Collaboration and Clinical Partners
This project, like so many, has resulted from the combined efforts of many individuals. The individuals noted above and in the acknowledgments have provided a great deal of encouragement and feedback that have been vital for the design and execution of this study.
As discussed elsewhere (McAleavey et al., 2015), partnerships between researchers and clinics are oftentimes complex for a wide variety of reasons. One lesson we have learned through engaging in these partnerships is the central importance of clinic director buy-in and commitment to supporting and encouraging research among their staff. As evidence for this, we began our recruitment efforts by contacting clinic directors at 15 clinics using the BHM. Of these, therapists from five clinics completed the measures, resulting in a sample of 17 participants. Committed to obtaining a sample of therapists that would allow adequate power to detect small to moderate associations between therapist characteristics and patient outcomes, we then expanded our recruitment by reaching out to a clinic director with whom we have formerly collaborated. This expansion has resulted in an additional 40 participants (Figure 1).
Data collection has likely been more successful since adding our collaborator’s site in part because it is a large clinic with many therapists. Many therapists at this center are in training and may, therefore, be more motivated by the compensation provided and able to devote two hours of their own time to complete our study. It also seems likely that recruitment has been successful due to the established partnership between the clinic director and our staff. This existing relationship may support the sense of shared mission and shared ownership that McAleavey et al. (2015) discussed as crucial for these partnerships.
With our goal of collecting a multimodal assessment battery with a variety of self-report and behavioral measures, a second equally important partnership has been with a technical team. This partnership has involved collaboration with developers of the performance tasks noted above. In addition, developing our assessment delivery platform was a protracted process, during which we received guidance from programmers with expertise in online behavioral tasks. We conducted several rounds of pilot testing of our platform and made a series of changes based on the feedback we received. We established a collaboration with Theravue.com in order to use their electronic deliberate practice platform for delivering the FIS and MCO online. While the two-hour battery remains a substantial time commitment, we have found that most participants are able to complete the assessments without technical issues. The piloting process delayed the initiation of data collection somewhat, but we are hopeful it will ultimately produce a larger sample, cleaner data, and a more robust scientific product.
The current study follows in a long tradition of psychotherapy research seeking to identify therapist characteristics that predict patient outcomes. This work aims both to replicate previous findings by evaluating measures that have been previously linked to patient outcomes (e.g., FIS; Anderson et al., 2009) and to extend this research by addressing several methodological shortcomings of the existing literature (e.g., small sample sizes, reliance on self-report measures, limited range of constructs assessed).
We hope that this study can help clarify what it is that makes effective therapists effective. In other words, we hope to find out what predicts the therapist effect. Of course, this is only an initial step towards the ultimate goal of reducing the burden of mental illness. Depending on what this study and similar efforts reveal, we may uncover certain capacities that could be selected for at the start of graduate training (e.g., through administering an interpersonal skills task as part of the admissions process). Other characteristics could be augmented through targeted training (e.g., mindfulness skills; Pereira et al., 2017) or built into training programs (e.g., deliberate practice; Chow et al., 2015). And yet other characteristics may interact with patient factors and could be used to support efforts to match therapists and patients, providing treatment that considers patient variability (i.e., precision medicine; Collins & Varmus, 2015).
A primary limitation of the current study is its correlational nature. Even if we are able to identify a clear set of therapist characteristics, they will simply be those associated with patient outcomes. Thus, a key future direction will be establishing a causal relationship between potentially malleable therapist characteristics and patient outcomes. Future randomized controlled trials could examine interventions aimed at these characteristics. Results from these experimental studies could then inform training efforts. This may be an ambitious goal, but certainly not outside the realm of possibility for the next decades of psychotherapy research. And, most importantly, these efforts may provide an empirically grounded route for improving the treatment outcomes and lives of our patients, during whatever challenging times the future may hold for human beings on this planet.
Author's Note: The author would like to thank the Society for the Advancement of Psychotherapy for support of this research through the 50th Anniversary Research Grant. The author is grateful for ongoing discussions with William Hoyt, Jesse Owen, Tony Rousmaniere, Zac Imel, David Atkins, Timothy Anderson, Betty VanLeuven, Bruce Wampold and others regarding this study and its design. The author is grateful to Wing Ng, Dan Fitch, and Robin Goldman for their assistance building the data collection platform and to Anthony Flynn and Ilsa Valdez for assisting with data collection. Correspondence regarding this article can be addressed to Simon B. Goldberg, Department of Counseling Psychology, 335 Education Building, 1000 Bascom Mall, University of Wisconsin, Madison, WI, 53706. Email: firstname.lastname@example.org
Cite This Article
Goldberg, S. (2020). Predicting the therapist effect: Study rationale and what we have learned without even looking at the data. Psychotherapy Bulletin, 55(2), 6-12.
Anderson, T., Ogles, B.M., Patterson, C.L., Lambert, M.J., & Vermeersch, D.A. (2009). Therapist effects: Facilitative interpersonal skills as a predictor of therapist success. Journal of Clinical Psychology, 65(7), 755-768. https://doi.org/10.1002/jclp.20583
Baldwin, S.A. & Imel, Z.E. (2013). Therapist effects: Findings and methods. In M.J. Lambert(Ed.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (6th ed.) (p. 258-297). Wiley & Sons.
Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., & Rubin,J. (in press). The psychological impact of quarantine and how to reduce it: rapid review of the evidence. The Lancet. https://doi.org/10.1016/S0140-6736(20)30460-8
Bui, Q., & Wolfers, J. (2020, March). More than 3 million Americans lost their jobs last week.See your state. The New York Times. https://www.nytimes.com/interactive/2020/03/26/upshot/coronavirus-millions-unemployment-claims.html
Chow, D., Miller, S. D., Seidel, J., Kane, R.T., & Thornton, J.A., & Andrews, W.P. (2015). The role of deliberate practice in the development of highly effective psychotherapists. Psychotherapy, 52(3), 337-345. https://doi.org/10.1037/pst0000015
Collins, F. S. & Varmus, H. (2015). A new initiative on precision medicine. New England Journal of Medicine, 372(9), 793-795. https://doi.org/10.1056/NEJMp1500523
Flückiger, C., Del Re, A.C., Wampold, B.E., & Horvath, A.O. (2018). The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4), 316-340. https://doi.org/10.1037/pst0000172
Garfield, S. L. & Bergin, A. E. (1971). Personal therapy, outcome and some therapist variables. Psychotherapy: Theory, Research and Practice, 8(3), 251-253.
Goldberg, S.B., Hoyt, W.T., Nissen-Lie, H., Nielsen, S.L., & Wampold, B.E. (2018). Unpacking the therapist effect: Impact of treatment length differs for high- and low-performing therapists. Psychotherapy Research, 28(4), 532-544. https://doi.org/10.1080/10503307.2016.1216625
Goldberg, S. B., Tanana, M., Imel, Z. E., Atkins, D. C., Hill, C. E., & Anderson, T. (in press). Can a computer detect interpersonal skills? Using machine learning to scale up the Facilitative Interpersonal Skills task. Psychotherapy Research. https://doi.org/10.1080/10503307.2020.1741047
Heinonen, E. & Nissen-Lie, H. A. (in press). The professional and personal characteristics of effective psychotherapists: a systematic review. Psychotherapy Research. https://doi.org/10.1080/10503307.2019.1620366
Johns, R. G., Barkham, M., Kellett, S., & Saxon, D. (2019). A systematic review of therapist effects: A critical narrative update and refinement to review. Clinical Psychology Review, 67, 78-93. https://doi.org/10.1016/j.cpr.2018.08.004
Jones, L., Brown, D., & Palumbo, D. (2020, March). Coronavirus: A visual guide to the economic impact. BBC. https://www.bbc.com/news/business-51706225
Kaplan, J., Frias, L., & McFall-Johnsen, M. (March, 2020). A third of the global population is on coronavirus lockdown – here’s our constantly updated list of countries and restrictions. Business Insider. https://www.businessinsider.com/countries-on-lockdown-coronavirus-italy-2020-3
Kopta, M., Owen, J., & Budge, S. (2015). Measuring psychotherapy outcomes with the Behavioral Health Measure-20: Efficient and comprehensive. Psychotherapy, 52(4). https://doi.org/10.1037/pst0000035
Lingiardi, V., Muzi, L., Tanzilli, A., & Carone, N. (2018). Do therapists’ subjective variables impact on psychodynamic psychotherapy outcomes? A systematic literature review. Clinical Psychology & Psychotherapy, 25(1), 85-101. https://doi.org/10.1002/cpp.2131
McAleavey, A. A., Lockard, A. J., Castonguay, L. G., Hayes, J. A., & Locke, B. D. (2015). Building a practice research network: Obstacles faced and lessons learned at the Center for Collegiate Mental Health. Psychotherapy Research, 25(1), 134-151. https://doi.org/10.1080/10503307.2014.883652
Mintz, J., O’Brien, C. P., & Luborsky, L. (1976). Predicting the outcome of psychotherapy for schizophrenics: Relative contributions of patient, therapist, and treatment characteristics. Archives of General Psychiatry, 33(10), 1183-1186.
Nissen-Lie, H.A., Monsen, J.T., Ulleberg, P., & Rønnestad, M.H. (2013). Psychotherapists’ self-reports of their interpersonal functioning and difficulties in practice as predictors of patient outcome. Psychotherapy Research, 23(1), 86-104. https://doi.org/10.1080/10503307.2012.735775
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660. https://doi.org/10.1177/1745691612462588
Owen, J., Haywood, S., & Drinane, J. (2018). Therapist multicultural orientation performance test: Understanding therapist clinical responses. Symposium presented at the Annual Conference of the American Psychological Association.
Pereira, J. A., Barkham, M., Kellett, S., & Saxon, D. (2017). The role of practitioner resilience and mindfulness in effective practice: A practice-based feasibility study. Administration and Policy in Mental Health and Mental Health Services Research, 44(5), 691-704. https://doi.org/10.1007/s10488-016-0747-0
Schöttke, H., Flückiger, C., Goldberg, S.B., Eversmann, J., & Lange, J. (2017). Predicting psychotherapy outcome based on therapist interpersonal skills: A five-year longitudinal study of a therapist assessment protocol. Psychotherapy Research, 27(6), 642-652. https://doi.org/10.1080/10503307.2015.1125546
Smith, M. & Glass, G. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760.
Swift, J.K. & Greenberg, R.P. (2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80(4), 547-559. https://doi.org/10.1037/a0028226
Tavernise, S. & Oppel, R. A. J. (2020, March). Spit on, yelled at, attacked: Chinese-Americans fear for their safety. The New York Times. https://www.nytimes.com/2020/03/23/us/chinese-coronavirus-racist-attacks.html
Thompson, M. N., Goldberg, S. B., & Nielsen, S. L. (2018). Patient financial distress and treatment outcomes in naturalistic psychotherapy. Journal of Counseling Psychology, 65(4), 523-530. https://doi.org/10.1037/cou0000264
Wampold, B. & Imel, Z.E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2nd ed.). Routledge. https://doi.org/10.4324/9780203582015
Wampold, B.E., Mondin, G.W., Moody, M., Stich, F., Benson, K., & Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “all must have prizes.” Psychological Bulletin, 122, 203–215. https://doi.org/10.1037/0033-2909.122.3.203
World Health Organization. (2020, March). Mental health and psychosocial considerations during the COVID-19 outbreak. WHO. https://www.who.int/docs/default-source/coronaviruse/mental-health-considerations.pdf