Web-only Feature

Web-only Feature

5 Lessons Learned Monitoring Psychotherapy Process and Outcomes

Evaluation Nightmare or Dynamic Dream?

A recent body of psychotherapy research is converging on the benefits of using client process and outcome feedback in clinical practice (Lambert & Shimokawa, 2011). Continual client feedback, also known as “routine outcomes monitoring,” “progress monitoring,” or “practice-based evidence,” refers to the collection of self-report data (e.g., symptoms, well-being, and the therapeutic alliance) from clients on a regular basis. Client feedback is interpreted against standardized metrics, enabling therapists to compare a client’s progress to an expected, or ideal, rate of change.

Regular client feedback has been found to improve therapy outcomes, both in individual (Reese, Toland, Slone, & Norsworthy, 2010) and couples counseling (Reese, Norsworthy, & Rowlands, 2012), due in large part to helping clinicians with early detection of clients who are regressing (Lambert & Shimokawa, 2011).

Moreover, receiving feedback is theorized to be a key ingredient (necessary though not sufficient) in the development of psychotherapist expertise (i.e., “greater skill with greater experience;” Tracey, Wampold, Lichtenberg, & Goodyear, 2014, p. 218-219). These findings have prompted many researchers—as well as The Society for the Advancement of Psychotherapy—to urge clinicians to incorporate outcome and alliance monitoring in routine practice (Laska, Gurma, & Wampold, 2014; Reese et al., 2010; Youn, Kraus, & Castonguay, 2012).

Benefits and Barriers to Progress Monitoring

The progress monitoring literature suggests that incorporating a client feedback system into psychotherapy practice facilitates continual learning by prompting therapist self-assessment. That is, when data is routinely collected and assessed, it acts as a system of checks and balances for one’s intuitions about treatment progress and alliance quality (Laska, Gurman, & Wampold, 2014).

However, embracing progress monitoring is perhaps easier said than done. For example, a recent practice research network study aimed at finding out “what clinicians want,” that is, what they would find relevant and helpful from psychotherapy research (Tasca et al., 2015), found that “progress monitoring” research was ranked near the bottom. That is, clinicians seemed to indicate that they did not consider progress monitoring research of high importance in their practice. Following up with a focus group, Tasca and colleagues found that the benefits of progress monitoring may not yet be widely known, and that the low ranking may reflect the disconnect between the research and practice of psychotherapy.

What Is It Like To Actively Monitor Progress?

Tasca and colleagues speculated that one way to address this research-practice gap in the psychotherapy community would be to encourage practitioners to test the “process of progress monitoring” on just a few cases to gauge for themselves whether the system would be logistically and clinically advantageous to incorporate into their practice (2015, p. 10).

To examine why researchers and practitioners are not yet connecting around progress monitoring, this article aims to lend narrative to some of the issues that exist in this gap through one graduate student’s experience applying progress monitoring to practice.

Lessons I Learned About Progress Monitoring

I learned how to incorporate progress monitoring into practice during my training at a site offering low-fee psychodynamic psychotherapy to community members. Specifically, this therapy group has integrated Partners for Change Outcome Management System (PCOMS) into the fabric of its practice. PCOMS (Duncan, 2012) consists of a 4-item outcome measure administered at the start of every session (asking the question “how have you been doing this week?” individually, interpersonally, socially, and overall) and a session rating measure administered at the end of the therapy hour (asking the question “how did the session feel today” in terms of the relationship, goals/topics, approach/method, and overall).

Clients indicate their rating on a continuous scale between low and high (PCOMS does not use specific anchors). Client data are entered into a secure online system that calculates a trajectory of expected progress, based on norms for the client’s age and gender, and initial self-assessment. When a client’s rating of outcome or alliance drops below a specific limit, the system encourages therapists to assess the situation and possibly change their approach. Regarding the quality of the alliance, therapists are encouraged to take note when the client’s session rating drops below a lower limit.

pcoms

In the spirit of contributing to the discussion around progress monitoring, and hopefully to spark more dialogue within the community of psychotherapy practitioners, here are five lessons I learned over the past 10 months of progress monitoring.

Lesson 1: Get Curious About My Resistance

Adding a new ingredient to one’s professional practice can be hard work. This is especially true for therapists, since our emotional responses during the transition must be examined and dealt with to minimize negative impact on our clients. So when I reflected on progress monitoring, I noticed that I felt some resistance. Part of me recognized my feeling as a natural response to change, especially since this particular change involved increased oversight of my work; to feel guarded when approaching progress monitoring is understandable.

I also felt something deeper going on, and I eventually identified how some of my resistance was linked to my theoretical orientation, and therein, related to my stance on larger philosophical questions about how research and practice interact. Identifying as a psychodynamic psychotherapist, I am drawn to the nuance and complexity of this theoretical orientation. It comes with the territory to struggle with how to apply research evidence to the uncontrolled environment of the therapy context, where subjective experience comes alive and practicing therapy can sometimes feel more like an art than a science.

At first glance, progress monitoring looks like it adds a heaping dose of “science” to practice. That is, progress monitoring is represented in terms of enhancing evidence-based practice in psychotherapy (Laska et al., 2014), but what is less clear is the type of evidence it adds to the mix. Examining evidence type may illuminate the issues that keep researchers and practitioners from connecting around progress monitoring. For the purpose of the present discussion (adapting from Rychlak, 1968), I define the evidence produced from progress monitoring as “objective,” because it asks practitioners to compare their clients’ progress to a norm, derived from a larger sample, and contextualizes evidence by bringing the group standard into focus. Contrast this with a more “subjective” point of view, which contextualizes evidence by bringing the individual’s experience into focus.

It is safe to say that most practitioners use both forms of evidence in their practice. They might base a clinical decision on a body of research when relevant themes arise in their work with a client. Just as likely (and perhaps in the very next moment) they may tune in to their own, or a client’s, subjective experience to guide their judgment about the most useful course of action.

I gravitate toward prioritizing a subjective evidence-base to oversee my clinical decisions. Though I consider empirical testing of psychodynamic theory and the gathering of objective evidence integral to the field (e.g., see Shedler, 2010), I am cautious when it comes to applying research very directly in the therapy context. My rationale centers around the fact that (put simply) research cannot necessarily be generalized from the group to the individual. Accordingly, in the therapy context, I think the therapist has a responsibility to tune in to her or his own subjective knowledge, along with the client’s, to derive truth, meaning, and evidence (Rychlak, 1968).

From this more subjective standpoint, I am most interested in my clients’ and my own interpretations of progress, change, and the therapy relationship. Correspondingly, one (subjective) evidence-based practice I use to oversee my clinical decisions is conversation with clients (Hill & Knox, 2009). That is, talking openly about how our unique perspectives of the work match up, how they diverge, and how the gap between our experiences can fuel insight and the direction we take in therapy.

These reflections on my resistance ultimately led me to ask:

  1. How does viewing progress monitoring as objective evidence influence how I use client feedback?
  2. As an evidence-based practice, what does progress monitoring add to my practice above and beyond the subjective methods I use to oversee my clinical decisions?

Lesson 2: Think About What Client Feedback Is Telling Me

Coming face to face with progress monitoring data implied grappling with a new, more objective, evidence-base to help oversee my clinical decisions. What does this actually look like? PCOMS prompts a therapist to ask “How is my client doing compared to other people?” and the answer to this question is displayed graphically, based on how clients assess their well-being from week to week (recall the questionnaire: “how have you been doing this week?” individually, interpersonally, socially, and overall).

For a scientist-practitioner, seeing this trajectory and gleaning its potential as a source of rich information can elicit a lot of excitement. Excitement is, in fact, a huge part of my experience of using PCOMS. At the same time, I faced the problem of having to settle on an interpretation of the data. That is, in order to open up a conversation about progress, or even recognize that it’s time for a conversation, the therapist needs to have a grasp on what the data mean. And just as important, the therapist also needs to make room for the client’s interpretation of the data. As I was acclimating to opening these conversations, I learned a humbling lesson about failing to make room for the client’s interpretation:

At the start of session four with one of my clients, I glanced at his progress score and saw that it had dropped considerably since we began therapy. Throughout the hour, my client was very open about his distress and symptoms. As I listened, I found myself remembering the progress monitoring literature’s suggestion that if a client is regressing, a change of course may be appropriate (Lambert & Shimokawa, 2011). Worried that therapy was causing my client to regress, I took on a directive approach. Speaking to the consistent drop in his well-being score, I said was concerned that he seemed to be getting worse since therapy started. I suggested that an assessment from a psychiatrist might help us get a clearer picture of what was going on and help guide the therapy. In the moment, my client seemed open to the idea and asked for some referrals, however, his rating of the session fell slightly from the previous week.

During the next session, I was surprised how he responded when I followed-up about his session rating. He reported feeling confused all week because my reaction, and the suggestion to see a psychiatrist, did not match what he felt inside. He explained his take on the data and how he was responding with greater honesty about his distress as therapy started to feel safer. Moreover, based on my reaction, he felt disappointed and angry in light of my misattunement. We worked through this miscommunication largely through making space for him to express his experience and feel heard.

Reflecting on my initial reaction in this case, I recognize now that it is wise to simply pause and ask clients what they think the data mean. This lesson gets to the heart of the question of what type of evidence progress monitoring brings to the mix. In other words, in this example, I was emboldened by what appeared to be an objective truth about my client. That is, (presumably) compared to everyone else, he was getting worse. Backed by objectivity, I felt license to take charge and barrel through without really listening.

To answer my first question, I found that it is key to remember that a progress monitoring system cannot lend purely objective evidence to a clinical decision. Therapists will use the data more effectively if they acknowledge their own subjectivity in interpreting client feedback.

Lesson 3: Don’t Evaluate...

The illusion that the evidence of progress monitoring is not purely objective is a tricky thing to shake off. Examining the ups and downs of a client’s progress, superimposed over the trajectory indicating where they “should” be, it was difficult at first to avoid comparison and self-judgment. Put another way, it was all too easy to look at the data and ask: “How am I performing with this client in relation to how other therapists might perform, on average?”

Though natural to ask, this question overestimates our ability to use progress monitoring for the purpose of objectively comparing one therapist’s skill against another’s. After all, the algorithm that calculates a client’s standardized trajectory does not take into account the severity of the client’s core issue, any personality factors that might make the client more resistant to change, or any existential themes wherein progress is actually measured in part by the client feeling worse for a while (e.g., see discussion in Hill, Chui, & Baumann, 2013). Additionally, how a given client changes, compared to the average, is necessarily based on the interaction of client factors and therapist factors. So if anything, self-assessment based on how far away a client is from the standard on a given week is really more like a projective test: the therapist will cast his or her sense of self-efficacy onto the data.

Lesson 4: … Instead, Calibrate

Perhaps especially for trainees, it is tempting to jump to the conclusion that they are doing objectively bad (or good) work. Introducing a standardized evidence-base amplifies this impulse. In all likelihood, addressing one’s self-judgment is a task that each practitioner must manage in his or her own way when beginning to use progress monitoring. Achieving this task is rewarding, however, because when progress monitoring begins to feel less like an evaluation, it becomes a valuable tool for growing a therapist’s interpersonal skills, modeling openness, and improving accuracy of empathic and relational attunement (note: areas of competence that align with the psychodynamic approach in particular).

For example, toward the end of the third session with one of my clients, I sensed that an unspoken rupture may have occurred between us however I was not completely certain. Typically when this happens, I will reflect on the evidence I have for the rupture and make a mental note to bring it up with the client at the start of our next session. However, over the week, self-doubt or skepticism sometimes overtakes my gut reaction about the best way to respond, reducing the likelihood that I will ask my client to process our relationship in the next session.

In this case, thanks to my client’s session rating data, my intuition about our rupture was instantly confirmed when I saw that her score indicated that the session did not feel helpful. Though it was difficult to see that she felt let down by the session (and after I took time to regulate my feelings about her feedback), I knew with certainty that the best clinical decision was to open up space to talk about at our next session. In other words, because of the multiple forms of evidence, I knew to approach the conflict and work on repairing the rupture. Essentially, progress monitoring added a safety net to my imperfect approach-avoid decision-making, ensuring that the client’s shift of experience in the dynamic would not go unprocessed.

To answer my second question, it became very clear that progress monitoring was useful above and beyond my subjective decision-making, and it was helping me improve as a practitioner. Over time, calibrating my clinical intuitions based on client feedback helped me grow in several ways.

A side effect of de-personalizing clients’ feedback—instead, reading it as an opening to adjust my clinical decisions—was a noticeable increase in my tolerance of clients’ negative emotion and an improvement in my ability to contain and address their difficult feelings. In turn, I have capitalized more on opportunities to bring corrective emotional and relational experiences into the therapy.

Additionally, I have become more open and transparent in testing my hypotheses, through both confirmation and disconfirmation (Tracey et al., 2014). Progress monitoring data cues me to reflect on the evidence I have for my intuitions. In turn, I tend to be more transparent with my clients about how I interpret their experiences, and my clients now have a tool prompting them to be transparent about their immediate thoughts and feelings. Regularly asking for feedback invites clients to place value on their inner experience, and I have found my clients increasing their tendency to look within themselves for their own answers.

Lesson 5: Be Authentic, Get Creative, Look Ahead

The takeaway of my experience with progress monitoring can be summarized thus far as 1) appreciating the subjective nature of progress monitoring data even though it is an evidence-based practice on the more objective end of the evidence spectrum, 2) recognizing how self-judgment limits the way progress monitoring contributes to professional growth, and 3) experiencing first-hand how adding client feedback helps me regulate my clinical intuition and improves both certainty and accuracy in my clinical decision-making process. Adding on to my personal narrative, several other key points that arose from the discussions I’ve had with colleagues are also worth speaking to.

First, adapting progress monitoring such that it fits each therapist’s style seems like it might be key to a successful integration. Similar to research suggesting that adhering closely to a therapy treatment manual does not necessarily improve outcomes (Webb, DeRubeis, & Barber, 2010), attempting to “manualize” progress monitoring may not be beneficial. Instead, figuring out how one will talk about client feedback, how often to review feedback, and when to show clients their progress are all choices that therapists ought to make based on what feels most authentic to their practice and preferences. For example, I find that it fits my style to introduce the feedback system as a way for clients to communicate with me. This framing seems to set the stage for clients to recognize and value the role of openness in therapy, but I believe it has this effect because it reflects my genuine opinion about how openness is fundamental to change.

Second, introducing progress monitoring to practice seems to produce a wealth of clinical information that practitioners, especially the psychodynamically-oriented among us, can draw from to form conceptualizations and intervention strategies. Getting creative in thinking about what a client may implicitly communicate to us can inspire innovative strategies. For example:

  • A client who narrates his or her session rating by saying they are leaving “room for improvement” may have a complicated relationship with positive reinforcement and may benefit from processing how it feels to receive praise and validation from their therapist.
  • A client who is showing progress but is prone to change-resistant, harsh self-evaluation may benefit from being able to visualize how they have changed over time, especially if his or her actual progress surpasses the expected trajectory.
  • A client who indicates high levels of well-being throughout the beginning stage of therapy, yet keeps returning, may be attending on the behest of someone else and could be prone to concealing their genuine feelings.

Third, talking about and implementing progress monitoring also presents constructive opportunities to look ahead and shape the future of our field. For example, at our practice, progress monitoring has lead to discussions of our impact as a whole on the community that we serve. Our ability to examine standardized metrics embedded in PCOMS, such as the approximate effect size of client improvement across the entire practice, has resulted in rich discussions about our role as change agents in our clients’ lives and the wider community. Evidence such as this and the resulting discussions may increase in importance as the health care system changes and as the public become increasingly savvy consumers of health data.

Finally, circling back around to the findings of Tasca and colleagues (2015), my experience using progress monitoring has increased my interest in how progress monitoring research could inform my practice. A few questions I have include:

  • How does progress monitoring influence therapy process?
  • What effect does progress monitoring have on time spent processing the therapy relationship?
  • Are there optimal moments to review progress with clients?

It is my hope that others in the psychotherapy community will feel compelled to continue the conversation about progress monitoring, and eventually see this topic as a rich, nuanced, even pivotal area that is relevant to both research and practice.

Cite This Article

Savela, A. (2015, August). 5 lessons learned monitoring psychotherapy process and outcomes: Evaluation nightmare or dynamic dream? [Web article]. Retrieved from: https://societyforpsychotherapy.org/5-lessons-learned-monitoring-psychotherapy-process-and-outcomes-evaluation-nightmare-or-dynamic-dream

References

Duncan, B. L. (2012). Partners for Change Outcome Management System (PCOMS): The heart and soul of change project. Canadian Psychology, 53(2), 93-104. doi: 10.1037/a0027762

Hill, C. E., Chui, H., & Baumann, E. (2013). Revisiting and reenvisioning the outcome problem in psychotherapy: An argument to include individualized and qualitative measurement. Psychotherapy, 50 (1), 68-76. doi: 10.1037/a0030571

Hill, C. E., & Knox, S. (2009). Processing the therapeutic relationship. Psychotherapy Research, 19(1), 13-29.

Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48(1), 72-79. doi: 10.1037/a0022238

Laska, K.M., Gurman, A. S., & Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467-481. doi: 10.1037/a0034332

Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy Theory, Research, Practice, Training, 46(4), 418-431. doi: 10.1037/a0017901

Reese, R. J., Toland, M. D., Slone, N. C., & Norsworthy, L. A. (2010). Effect of client feedback on couple psychotherapy outcomes. Psychotherapy Theory, Research, Practice, Training, 47(4), 616-630. doi: 10.1037/a0021182

Rychlak, J. F. (1968). Methods of evidence in arriving at knowledge. In A philosophy of science for personality theory (73-95). Boston, MA: Houghton, Mifflin and Company.

Shedler, J. (2010). The efficacy of psychodynamic psychotherapy. American Psychologist, 65(2), 98-109. doi: 10.1037/a0018378

Tasca, G. A., Sylvestre, J., Balfour, L., Chyurlia, L., Evans, J., Fortin-Langelier, B., . . .Wilson, B. (2015). What clinicians want: Findings from a psychotherapy practice research network survey. Psychotherapy, 52(1), 1-11. doi: 10.1037/a0038252

Tracey ,T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69(3), 218-229. doi: 10.1037/a0035099

Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200-211. doi: 10.1037/a0018912

Youn, S. J., Kraus, D. R., & Castonguay, L. G. (2012). The treatment outcome package: Facilitating practice and clinically relevant research. Psychotherapy, 49(2), 115-122. doi: 10.1037/a0027932

0 Comments

Submit a Comment

Your email address will not be published.