Web-only Feature

Web-only Feature

Want to Know Your Blind Spots? Ask Your Clients!

A Feedback-Informed Primer

Internet Editor’s Note: Dr. Sean Woodland and colleagues recently published an article titled “Do clinicians really use feedback-monitoring systems? A qualitative analysis of 16 group leaders” in Psychotherapy.

If you’re a member of the Society for the Advancement of Psychotherapy you can access the Psychotherapy article via your APA member page.

Not a member? Purchase the Psychotherapy article for $11.95 here.

While there is little debate about whether psychotherapy works, there remains disagreement about how it works (Barlow, 2004; Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2014; Lorenzo-Luaces, German, & DeDubeis, 2014; Messer & Wampold, 2002; Shafran et al., 2009; Tracey, Wampold, Lichtenberg, & Goodyear, 2014; Wampold, 2015).  This lack of consensus has understandably led to continued questions, frustration, and calls for unification to drive the field forward (Mulder, Murray, & Rucklidge, 2017).   One specific way in which the field has sought consensus is through feedback-informed care, characterized by using routine monitoring systems (ROMs).  Designed to be transdiagnostic and transtheoretical, the focus of these systems is on psychotherapy outcomes, and on alerting therapists about clients who may be at risk for treatment failure.  Many studies have demonstrated the incremental benefit of using such outcome monitoring systems (Kluger & DeNisi, 1996; Krägeloh, Czuba, Billington, Kersten, & Siegert, 2015; Sapyta, Riemer, & Bickman, 2005).  However, as these and other systems are developed and refined, new questions arise about how/in what ways using feedback leads to this difference in outcome.

Why Feedback is Important

The main truism that feedback-informed care leans on is the idea that therapists are not as good at predicting a client’s status as perhaps we think we are.  In fact, several studies have shown that therapists poorly predict which clients will improve, which will stay the same, and which will get worse at the end of a treatment episode (Breslin, Sobell, Buchan, & Cunningham, 1997; Chapman et al., 2012; Hannan et al., 2005; Norcross, 2003).  In general, therapists tend to over-predict their clients’ outcomes, and over-predict their own effectiveness (Walfish et al., 2012).  While therapist optimism about the course of treatment in many cases may be appropriate to frame expectancy, it seems that therapists are missing a great deal of information regarding the client’s clinical status.  Michael Lambert (2013), a key figure in the development of ROMs, noted:

With self-assessments that are this far inflated, and confidence in their ability to predict treatment outcome and the therapeutic alliance, therapists are not rushing to use [outcome monitoring systems], just as there is a reluctance to adopt other evidence-based practices. (p. 204)

While Lambert’s assertion is striking, it is amplified further by the suggestion that roughly half of psychotherapy outcomes are attributable to factors related to the client (Lambert, 1992; similarly, Cuijpers et al. 2012).  If we control for the skill of the therapist, significant improvement of client distress is perhaps just as determinable by the flip of a coin as by the intervention, skill, experience and theoretical orientation of the therapist. While this constitutes a major blind spot, ROMs are specifically designed to shine light where we tend to not see, or where we may choose not to look.

A Recent Application

While the evidence base for ROMs increases, new questions arise regarding how the use of feedback influences outcome.  With this in mind, Whitcomb, Woodland, and Burlingame (2018) sought to find out more about how group therapists used the measure-based feedback they received.  Two separate qualitative content analyses were conducted based on data provided by 16 group therapists across 29 therapy groups.  The feedback came from the Group Questionnaire (GQ; Janis, Burlingame, & Olsen, 2018), which provides information regarding the client’s perception of the bond, work, and conflict in the therapy group.  Data regarding how the leader used the feedback were collected each week via open-ended prompts, and at the end of the treatment episode via debrief interviews.  The two studies yielded a total of 3,364 parsed units of analysis.

The results from the two content analyses mirrored each other, such that the authors determined with some confidence that there is a common pattern for GQ feedback use.  The common themes included using the feedback prior to the start of the group (e.g. reacting to the results, planning interventions), use during the group (e.g., speaking about certain results in the group, noticing group members’ reaction to these conversations), and use after the end of the group session (e.g., effect on the group moving forward, opinions about the feedback in hindsight).

Once a common coding frame was developed, the authors then sought to find whether there were statistically significant predictors of GQ use.  They found group leaders tended to vary in ways they used the feedback; for example, some leaders tended toward using the feedback before the group session, while others preferred to use the feedback in-vivo with group members/clients.  It was also determined that the group itself was a significant source of variability.  And, interestingly, this variability of GQ use was observed even between groups led by the same leader.  The authors concluded that leaders used the GQ feedback differently based on the makeup of the therapy group, an assertion which was also supported by unsolicited comments from the group leaders themselves.  The authors then used the data to create a questionnaire to assess use of the GQ, which in their study showed good reliability and criterion validity.

One important omnibus conclusion was that due to therapist differences in feedback use, the relationship between feedback and outcome in groups cannot be intuited linearly; that is, moderators must also be considered.  While feedback studies in individual therapy have shown some evidence of moderation, (Bickman et al., 2011; de Jong, van Sluis, Nugter, Heiser, & Spinhoven, 2012; Durham et al., 2002), significant moderators of this relationship in group psychotherapy have yet to be identified (Burlingame et al., 2018).

A Place to Intervene

As future research on ROMs continues to elucidate how their use leads to client change, it’s important to recognize the important parallel of clinician training in using the data most effectively.  While clinicians’ varying styles may dictate different emphases within the overall implementation and value of feedback (Whitcomb, Woodland, & Burlingame, 2018; Woodland, 2015), below are some common threads intended to guide clinicians.  All three suggestions involve the clinician acting upon the feedback in some way, either through transparent or indirect in-session discussions about the feedback.  Conversations should be open explorations of the data the client is providing and should be communicated as an opportunity for the clinician to learn, grow, and adapt to the client’s treatment needs.

Focus on who is in distress

The research on the feedback-outcome relationship in individual psychotherapy suggests that those who are most in distress perhaps have the greatest potential for change (Hawkins et al., 2004, Slade et al., 2008; Whipple et al., 2003).  Group therapists also reported success in focusing on those who were experiencing more relationship distress (Whitcomb, Woodland, & Burlingame, 2018).  Many ROMs provide feedback to clinicians regarding which clients have drifted across cut points indicating greater severity of symptoms, or what is considered “normal range.”  These alerts help clinicians narrow their therapeutic focus on those who perhaps need more attention than clients who are faring better in treatment.

Focus on who is at risk for treatment failure

Some ROMs (including the OQ-Analyst) specify which clients are in distress and who seem to not be progressing.  These clients are often considered “at risk for treatment failure,” and are intended through use of feedback to be priorities for clinical intervention.  This is determined usually by filtering those whose scores are beyond the normal range, and then by those whose scores have significantly worsened since the previous session.  This significant change is calculated with the aid of a reliable change index (RCI). In some cases, “at risk” clients are also determined when the client is not progressing consistent with an expected recovery curve (Lambert, 2015).  Suggested interventions may include approaching the client regarding how they feel treatment is progressing, which may be through conversation, administering supplemental measures, or both (Krägeloh, Czuba, Billington, Kersten, & Siegert, 2015).

Focus on the client’s subjective experience/therapeutic alliance

In addition to feedback on the client’s distress, clinicians also find feedback on the therapeutic alliance to be useful (Hawkins et al., 2004, Slade et al., 2008; Whipple et al., 2003; J. Peters, personal communication, November 15, 2018; Whitcomb, Woodland, & Burlingame, 2018).  This feedback might be directed toward potential alliance ruptures, or simply how the client is perceiving treatment progress.  Many measures have been developed and validated which assess the therapeutic alliance, including the GQ mentioned here.  For group therapists, the GQ measures the alliance across member-leader, member-group, and member-member domains, providing rich feedback across the complexities of group interactions.  The intended use of the GQ, along with other measures of the alliance used in feedback systems, is to instruct and provide the clinician greater opportunity to process and further strengthen the alliance, thus providing the client greater opportunity for growth.


  • Routine outcome monitoring systems (ROMs) continue to develop and garner support in both the research and clinical arenas.
  • They can provide the psychotherapist with rich detail about the client’s subjective experience which may not be readily visible during therapy. This is relevant to clients who may be at risk for treatment failure.
  • Treatment outcomes also improve when clinicians use the feedback that ROMs provide.
  • Very early returns suggest variability in how clinicians utilize the feedback and warrants future exploration in both the individual and group arenas.


Sean Woodland, PhD is a psychologist working at Kaiser Permanente Stockton, CA Medical Offices. His areas of focus are in feedback-informed care, behavioral medicine, psychopharmacology, group treatment, and organizational leadership/development. He maintains affiliations with APA's Division 49 (Group Psychology and Group Psychotherapy) and Division 38 (Health Psychology).

Cite This Article

Woodland, S. (2019, February). Want to know your blind spots? Ask your clients! A feedback-informed primer. [Web article]. Retrieved from http://www.societyforpsychotherapy.org/want-to-know-your-blind-spots-ask-your-clients


Barlow, D. H. (2004). Psychological treatments. American psychologist59(9), 869.

Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services62(12), 1423-1429.

Breslin, F., Sobell, L. C., Buchan, G., & Cunningham, J. (1997). Toward a stepped-care approach to treating problem drinkers: The predictive validity of within-treatment variables and therapist prognostic ratings. Addiction, 92(11), 1479-1489.

Burlingame, G. M., Whitcomb, K. E., Woodland, S. C., Olsen, J. A., Beecher, M., & Gleave, R. (2018). The effects of relationship and progress feedback in group psychotherapy using the Group Questionnaire and Outcome Questionnaire–45: A randomized clinical trial. Psychotherapy55(2), 116.

Chapman, C. L., Burlingame, G. M., Gleave, R., Rees, F., Beecher, M., & Porter, G. S. (2012). Clinical prediction in group psychotherapy. Psychotherapy Research22(6), 673-681.

Cuijpers, P., Driessen, E., Hollon, S. D., van Oppen, P., Barth, J., & Andersson, G. (2012). The efficacy of non-directive supportive therapy for adult depression: a meta-analysis. Clinical psychology review32(4), 280-291.

de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy research22(4), 464-474.

Durham, C. J., McGrath, L. D., Burlingame, G. M., Schaalje, G. B., Lambert, M. J., & Davies, R. D. (2002). The effects of repeated administrations on self-report and parent-report scales. Journal of Psychoeducational Assessment, 20, 240 –257. http://dx.doi.org/10.1177/073428290202000302

Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of clinical psychology61(2), 155-163.

Hawkins, E., Lambert, M., Vermeersch, D., Slade, K., & Tuttle, K. (2004). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14, 308–327. http://dx.doi.org/10.1093/ptr/kph027

Janis, R. A., Burlingame, G. M., & Olsen, J. A. (2018). Developing a therapeutic relationship monitoring system for group treatment. Psychotherapy55(2), 105.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin119(2), 254.

Krägeloh, C. U., Czuba, K. J., Billington, D. R., Kersten, P., & Siegert, R. J. (2015). Using feedback from patient-reported outcome measures in mental health services: a scoping study and typology. Psychiatric Services66(3), 224-241.

Lambert, M. J. (1992). Psychotherapy outcome research: Implications for integrative and eclectic therapists. In J. C. Norcross & M. R. Goldfried (Eds.) Handbook of Psychotherapy Integration (pp. 94-129). New York, NY: Basic Books.

Lambert, M. J. (2013). The efficacy and effectiveness of psychotherapy. In M. J. Lambert (Ed.) Bergin & Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 169–218). New York, NY: Wiley.

Lambert, M. J. (2015). Progress feedback and the OQ-System: The past and the future. Psychotherapy, 52, 381–390. http://dx.doi.org/10.1037/pst0000027

Lilienfeld, S. O., Ritschel, L. A., Lynn, S. J., Cautin, R. L., & Latzman, R. D. (2014). Why ineffective psychotherapies appear to work: A taxonomy of causes of spurious therapeutic effectiveness. Perspectives on Psychological Science9(4), 355-387.

Mulder, R., Murray, G., & Rucklidge, J. (2017). Common versus specific factors in psychotherapy: opening the black box. The lancet psychiatry, 4(12), 953-962.

Norcross, J. C. (2003). Empirically supported therapy relationships. In J. C. Norcross (Ed.), Psychotherapy relationships that work (pp.3-16). New York: Oxford University Press.

Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of clinical psychology61(2), 145-153.

Shafran, R., Clark, D. M., Fairburn, C. G., Arntz, A., Barlow, D. H., Ehlers, A., … & Salkovskis, P. M. (2009). Mind the gap: Improving the dissemination of CBT. Behaviour research and therapy47(11), 902-909.

Slade, K., Lambert, M. J., Harmon, S. C., Smart, D. W., & Bailey, R. (2008). Improving psychotherapy outcome: The use of immediate electronic feedback and revised clinical support tools. Clinical Psychology and Psychotherapy, 15, 287–303. http://dx.doi.org/10.1002/cpp.594

Tracey, T. J., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal?. American Psychologist69(3), 218.

Walfish, S., McAlister, B., O’donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports110(2), 639-644.

Wampold, B. E. (2015). How important are the common factors in psychotherapy? An update. World Psychiatry14(3), 270-277.

Whipple, J., Lambert, M., Vermeersch, D., Smart, D., Nielsen, S., & Hawkins, E. (2003). Improving the effects of psychotherapy: The use of early identification of treatment failure and problem solving strategies in routine practice. Journal of Counseling Psychology, 50, 59–68. http://dx.doi.org/10.1037/0022-0167.50.1.59

Whitcomb, K. E., Woodland, S. C., & Burlingame, G. M. (2018). Do clinicians really use feedback monitoring systems? A qualitative analysis of 16 group leaders. Psychotherapy, 55, 132–143.

Woodland, S. C. (2015). Process Feedback in Group Psychotherapy: A Qualitative Inquiry into Leader Implementation of GQ/OQ Feedback (Unpublished doctoral dissertation). Brigham Young University, Utah.


Submit a Comment

Your email address will not be published. Required fields are marked *