We wrote an article entitled “Expertise in psychotherapy: An elusive goal” (Tracey, Wampold, Lichtenberg, & Goodyear, 2014), which has garnered several written (Hendlin, 2014; McMahan, 2014; Oddli, Halvorsen, & Rønnestad, 2014; Shanteau & Weiss, 2014) and oral reactions. We take this opportunity to clarify and expand our view. In our article, we argued that there is no evidence of expertise within the professional practice of psychotherapy. We focused our discussion on the profession itself, arguing that there was no relation between time spent in professional practice and improved practice. Among professional therapists, there is common lore that experience results in greater clinical “wisdom” and better skill. We found no support for this view with respect to clinical decision-making or clinical outcomes.
While this is a regrettable state of affairs, we do not see it as inherent to the profession. We then proposed that expertise can be developed with explicit feedback of clinical outcome in several ways: within clients over time (“Is this client improving relative to last session?”), across clients (“Is this client doing better than my other clients?”), and across therapists (“How do the outcomes that I obtain compare to those of other therapists?”). While we view this feedback as necessary, we do not view it as sufficient. There needs to be a reflective practice that accompanies clinical work and that such practice should manifest explicit hypothesis testing in a disconfirmatory manner, whereby common heuristics are minimized.
Thus our arguments focused on the profession rather than on the identification of expertise in individuals. Hendlin (2014) includes both definitions (profession and individual) when he takes umbrage at the implication that there are no therapists with expertise. There may indeed be individuals who have expertise but such a determination is different from our focus in the article. The issue of the determination of individual expertise is an interesting one, though, and merits review to clarify where it differs from and overlaps with the issue of the expertise of the profession.
Defining Individual Expertise in Psychotherapy
A therapist’s expertise has been variously defined or understood in terms of (a) reputation, degree attainment, professional distinction, and experience, (b) skill, competence, or adherence to a prescribed standard of performance, (d) clinical accuracy, or (d) outcomes—that is, success with clients. We will review each in turn.
Reputation, degree, professional distinction, or experience as expertise criterion
Reputation as an expert often is employed as a criterion for expertise. In studies of expert performance in psychotherapy, researchers often use nominations of peers (see, e.g., Burlingame & Barlow, 1996; Linehan et al., 2006; Jennings & Skovholt, 1999Skovolt & Jennings, 2004, as reputation is presumably derived from both skills and outcomes. Overholser (2010), in his attempt to identify the core elements of clinical expertise in psychotherapy, indicated that for a professional to be an expert “the professional must possess a terminal degree in the field… [and] the professional is visible in the professional community at a national level” (p. 131). Unfortunately, reputation, degrees, and professional recognition may not relate to performance. For example, therapists others view as experts have been found to fare no better than non-experts in the accuracy of their decisions (Faust, 1991; Faust & Ziskin, 1988; Garb, 1998, 2005; Witteman, Weiss, & Metzmacher, 2013).
Professional experience also has been used in definitions of expertise. Overholser (2010) defines the expert, in part, as a “professional [who] has accumulated multiple years of clinical experience in the direct provision of clinical assessment, psychological testing, or psychological treatment” (p. 131). We noted that there is little evidence of the superiority of more experienced therapists (e.g., Berman & Norton, 1985; Beutler, 1997; Beutler et al., 2004; Budge et al., 2013; Hattie, Sharpley, & Rogers, 1984; Laska, Smith, Wisclocki, & Wampold, 2013; Minami et al., 2008; Okiishi et al., 2003; Stein & Lambert, 1984, 1995; Strupp & Hadley, 1979; Wampold & Brown, 2005). So expertise definitions that include reputation, degree attainment, and experience are insufficient.
Task performance as expertise criterion
Many definitions of expertise stipulate that experts are those who perform various tasks with a high level of skill. Overholser (2010) made such a stipulation: “The [expert] professional has demonstrated evidence of superior clinical skills in a specific application of psychology” (p. 131). Research on aspects of psychotherapy often identifies experts according to these criteria, such as using psychologists who have been awarded diplomate status by the American Board of Professional Psychology (e.g., Eugster & Wampold, 1996; Tracey, Lichtenberg, Goodyear, Claiborn, & Wampold, 2003). Of course, this is general and others have been more specific, usually referring to the core skills as competencies or the psychotherapist who has the skills as being competent. This definition of expertise, that of matching standards of practice, is also advocated by Shanteau and Weiss (2014) in their comment on our article.
Another example would be those who advocate specific treatments and emphasize that expertise depends selecting and administering the optimal evidence-based treatment protocol (Waltz, Addis, Koerner, & Jacobson, 1993). Most evidence-based treatments define what constitutes competence within such treatments. For example, Rector and Cassin (2010) have recently outlined the competencies necessary for expertise in delivering cognitive-behavioral treatments
The identification of core competencies of professional practice (e.g., Kaslow, 2004) provides criteria against which to assess therapist behavior. But these competencies are important for understanding expertise only if those who have these competencies demonstrate levels of performance greater than those with only basic levels of competence. That is, are therapists who possess the core competencies, or have higher levels of such competencies, more effective than others? Shanteau and Weiss (2014) note the inherent problem with defining expertise as adherence to established standards of practice using the example of expertise in astrology. While one may indeed be an expert with respect to adhering to astrology standards, one is no more accurate. When discussing the competencies, Kaslow (2004) noted,
Competencies are composed of knowledge, skills, and attitudes, which, as a coherent group, are necessary for professional practice. They correlate with performance, can be evaluated against well accepted standards, and can be enhanced through training and development. (p. 775, emphasis added)
However, the literature that attempts to identify core competencies rarely, if ever, relies on evidence that the competencies are causally, or even correlationally, related to outcomes. It appears that (a) experts’ ratings of competence after observing therapists’ performance with clients do not predict outcome in general (Webb, DeRubeis, & Barber, 2010), although they did find some support for the treatment of depression (Wampold & Imel, 2015); (b) adherence to clinical protocols (i.e., degree to which the therapist follows the treatment manual) appears to be unrelated to outcome (Webb et al., 2010; Wampold & Imel, 2015); and (c) strict adherence to those protocols might even attenuate therapeutic outcomes (Castonguay, Goldfried, Wiser, Raue, & Hayes, 1996; Henry, Strupp, Butler, Schacht, & Binder, 1993). So there is little basis for the validity of using task performance as a criterion of expertise.
Clinical accuracy as expertise criterion
Much of the research on expertise in psychotherapy has concerned therapists’ cognitive accuracy in certain clinical task domains—specifically, their clinical judgment, clinical decision making, and case formulations (Betan & Binder, 2010; Eells et al., 2005; Faust, 2007; Garb, 1989; Spengler et al., 2009; Wierzbicki, 1993)—and there is little support that skill increases with greater experience. For example, Witteman et al. (2012) found that master’s level students were better at diagnosing depression than experienced therapists. In fact, Shanteau’s (1992) determination that clinical psychologists do not demonstrate expertise was made in part on the basis of the accuracy of their clinical judgment. Perhaps it is relevant as well that mechanical, actuarial approaches to prediction consistently outperform psychologists’ clinical judgments (Meehl, 1954; 1986), though the extent to which this is true depends on the type of data being employed in prediction (Grove, Zald, Lebow, Snitz & Nelson, 2000). Despite the lack of evidence regarding the relation of experience and clinical decision making, there has never been a demonstration that more accurate clinical judgments are related to better outcomes in psychotherapy. While this connection appears logical and necessary, it still remains to be empirically evaluated.
Clinical judgment and decision making also has been discussed with respect to experts’ thinking processes. That is, experts should perform cognitive tasks differently than novices (Bereiter & Scardamalia, 1986), at least with respect to reasoning about and solving domain-relevant problems. Oddli et al. (2014), in their comment on our article, focus on this by noting that there is a body of literature that demonstrates that more experienced clinicians think differently from less experienced clinicians (e.g., Huppert et al., 2001; Kivlighan & Quigley, 1991; Martin et al, 1989; Miller, Hubble & Duncan, 2008; Ronnestad & Skovholt, 2013). While this may be true, there is no direct connection between the different processes used and better outcome. Indeed, with respect to expertise in general, these cognitive differences can cause non-experts to match or out-perform experts (e.g., Ericsson & Lehmann, 1996). Given this, any definition of individual expertise needs to incorporate information on actual performance as indicated in outcomes.
Outcome as expertise criterion
Client outcome might seem the ultimate definition of expertise. Indeed, it would seem curious to claim expertise for any therapist who did not reliably produce exemplary client outcomes. The classic expertise research on chess masters (Chi, 2006) identified those individuals who were extremely skilled in the domain (i.e., those who had long histories of winning) and then examined how these winning individuals approached the game and thought about it differently from those who were not as skilled. Hence strong outcomes are essential in any definition of expertise as it is a means of identifying those who are skilled.
Achieving consistently better outcomes across a range of clients is one means of defining expert therapists (Wampold & Brown, 2005). Baldwin, Wampold and Imel (2007) found that therapists who generally have better outcomes also are better able to form alliances with their clients. However, recent research (Budge et al., 2013) has found that, even using client outcomes, experienced psychologists actually perform worse than intern/postdocs. Based on a longitudinal study of 173 therapists treating over 6000 clients over a period of an average of 5 years, it appears that over time therapists gradually achieve poorer outcomes (Goldberg et al., submitted). Thus there appears to be little relation between professional experience and increased skill when client outcome is used as the basis of skill definition.
But there are difficult definitional issues even with outcome as the expertise criterion. It is recognized, for example, that outcomes of psychotherapy are due in large part to client variables, including severity of dysfunction, diagnosis, motivation (e.g., stage of change), social support, and resources (e.g., Bohart & Tallman, 2010; Groth‐Marnat, Roberts, & Beutler, 2001). So the issue of how to use outcome information is important. Caseloads vary in client difficulty (e.g., more experienced clinicians often carry more difficult clients), so any mean outcome may not take this into account. Caseloads will also vary by predominant diagnoses. Can outcome scores be compared across clients with depression, personality disorders, and substance abuse?
The possibility of needing separate outcome cutoffs for each diagnostic grouping may seem warranted but this also makes it difficult to make determinations of who is an expert. The metric of outcome varies. This could be taken to imply that expertise should only be determined within diagnostic grouping such that one would get a mean outcome score for work with clients with a diagnosis of depression (e.g., the work Minami et al., 2008, who focused on benchmarks for depression outcomes) and perhaps another one with clients with a diagnosis of borderline personality disorder. Of course, the metric could be standardized such that comparisons can be made (or aggregated) across different diagnostic groups, although such a procedure would obviously require an adequate set of base rates against which to compare the outcome of any individual’s caseload. Unfortunately, there are no such norms of group outcome readily available to clinicians, much less quality information on the outcomes of individual clinicians themselves.
In conclusion, there are issues with any definition of individual expertise in psychotherapy. These are emblematic of the difficulties of the development of expertise: If it can’t be defined, then it is difficult to say when it has been attained. But with respect to any definition of expertise, we see actual outcomes as a necessary ingredient and more work is needed to specifically examine expertise at both the individual and profession level.
Expertise of the profession of psychotherapy
We are concerned about the lack of empirical demonstration of expertise in the profession as a whole. Hendlin (2014) expressed outrage that we argued that there is no empirical support for expertise. He felt that our comments were insulting and erroneous and that better research is needed to demonstrate that experience leads to expertise. We agree that more research is needed but differ in that we do not dismiss the research that exists. Shanteau and Weiss (2014) offered two criteria for establishing expertise: coherence and correspondence. Coherence is the agreement with theory/common practice; correspondence is the agreement with an external reality. Psychotherapy would demonstrate expertise if the coherence criterion was adopted, according to Shanteau and Weiss. It is probable that over time clinicians become more aligned with prevailing practice norms or theories. The field fails, however, with the lack of demonstration of correspondence: We have not demonstrated the relation of professional experience with increases in outcomes (the external reality). While agreeing that psychotherapy does not demonstrate expertise using the correspondence definition, Shanteau and Weiss believe that the adoption of the coherence criterion may be appropriate for our profession.
On this we disagree strongly. In our opinion, the correspondence criterion is the only appropriate one and we should use clinical outcomes as the external reality/standard against which to evaluate expertness. At the same time we do not think that the failure to demonstrate this correspondence relation to date must be accepted as an established fact. In this regard, we made explicit arguments for feedback elements that could (and should) be incorporated into practice to enhance the development of expertise of the profession as defined by correspondence with outcome. And we argued further that a lack of quality feedback about what is done in therapy and its clinical outcomes and how such feedback information is used are the sources of limits to establishing expertise in psychotherapy.
We posited that for the attainment of expertise in psychotherapy it is necessary for therapists to obtain quality information about outcomes. It is still relatively atypical for therapists to obtain quality psychometrically supported information regarding their work with clients. With respect to distal outcomes, such as assessments at termination and 6 months to one year later, assessments are quite rare. As a result, therapists have little knowledge (other than perhaps if the client returns in the future) of eventual outcomes and thus there is no corrective information on impressions gleaned at termination. More information on distal outcomes is essential both with respect to within-therapist comparisons (which are one’s best and worst cases) but also with respect to normative comparisons (how are one’s outcomes relative to those of others?). Such information is rarely provided but is crucial if there is to be any learning from experience.
Furthermore, proximate, session by session information is also important (e.g., symptomology prior to each session and alliance assessments after each session). While there are some good programmatic efforts to study the provision of outcome feedback (both proximate and distal) to the therapist (Duncan, Miller, Wampold, & Hubble, 2010; Lambert et al., 2005; Lambert & Shimokawa, 2011; Miller, Duncan, & Hubble, 2005; Reese, Norsworthy, & Rowlands, 2009), this is just beginning and not yet widely adopted. And while the provision of such feedback is related to better outcomes, it has not been shown to lead to better outcomes over time for a given therapist. As a consequence, although quality outcome feedback (both proximal and distal) is necessary for the development of expertise, we do not see it as sufficient.
One of the findings from the expertise literature is that expertise develops with deliberate practice (Ellis, Carette, Anseel & Lievens, 2014; Ericsson, 2006, 2009). It has been argued (e.g., Miller, Duncan, Sorrell, & Brown, 2005; Miller, Hubble, Chow, & Seidel, 2013) that dedicated time should be set aside for the examination of practice. In this regard, Chow et al. (in press) have found that whereas experience itself did not predict outcomes, the amount of time spent targeted at improving therapeutic skills did. However, the specifics of what should be incorporated into this deliberate practice aimed at improving skills are vague. We specified several aspects that should characterize this deliberate practice (i.e., adopting a disconfirmatory approach, explicitly testing hypotheses, and avoiding heuristics).
Incorporation of these strategies into deliberate practice should increase the chances of learning from the feedback embedded in outcome information. Our article focused on the lack of relation of experience and expertise and possible mechanisms to ameliorate this. Specifically, we called for much more quality outcome information (proximal and distal) and suggested that conscious attempts to use this information in deliberate ways (e.g., minimizing heuristics) would increase the relation of expertise and experience.
Some reactions to our article focused on practices that might lead to skill acquisition that in turn would lead to expertise. McMahan (2014) noted that in our paper we ignored supervision, which is the most widely used mode of improving practice. We see supervision as an important vehicle for obtaining and processing feedback (a key issue). But most supervision is performed while in training and our comments focused on development after training where there is much less supervision provided. In addition, whereas there is some evidence that supervision positively affects client outcome (Bambling, King, Raue, Schweitzer & Lambert, 2006) there also is recent research showing that there are no differences in client outcome across supervisor (Rousmaniere, Swift, Babins-Wagner, Whipple, & Berzins, 2014). Perhaps the issue should not be one of increasing supervision post degree but rather of examining the key elements within supervision that lead to better outcomes. It is our assumption that these too will be similar to our components of deliberate practice.
Oddli et al. (2014) viewed our recommendations for deliberate practice as a bit too narrow in that they would exclude implicit reasoning and intuitive practice decisions. We agree that these are subtle and potentially important aspects of cognitive development; but, without specifying new ways that these can be increased or honed, it is an example of proposing exactly what we already have. How specifically can these context-dependent decisions be enhanced and increased, and would this lead to better outcomes? Further, it is crucial that these intuitive decisions be separated from the common heuristics associated with such processing, because the errors of such processes have been well demonstrated (e.g., Dawes, 1994; Garb, 1997, 1998, 2005). There is some emerging research in this area of logical intuition (e.g., De Neys, 2012) but its application to psychotherapy awaits.
Admittedly psychotherapy is a difficult profession in which there are no clear answers and for which the definition and identification of expertise may be contentious. But we see our practices as open to improvement with careful attention and empirical evaluation. Stronger research-based attempts to help ensure that experience is used appropriately and results in increased skill and improved outcomes are needed. Our recommendations are in line with this view. That said, we do not think that ours are the only potential solutions to gaining expertise as a therapist, but believe that they are promising given the empirical base garnered from other areas.
Cite This Article
Tracey, T. J. G., Wampold, B. E., Goodyear, R. K., & Lichtenberg, J. W. (2015). Improving expertise in psychotherapy. Psychotherapy Bulletin, 50(1), 7-13.
Baldwin, S. A., Wampold, B. E., & Imel, Z. E. (2007). Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75, 842-852. DOI: 10.1037/0022-006X.75.6.842
Bambling, M., King, R., Raue, P., Schweitzer, R., & Lambert, W. (2006). Clinical supervision: Its influence on client-rated working alliance and client symptom reduction in the brief treatment of major depression. Psychotherapy Research, 16(3), 317-331. DOI: 10.1080/10503300500268524
Bereiter, C., & Scardamalia, M. (1986). Educational relevance of the study of expertise. Interchange, 17, 10-19. DOI: 10.1007/BF01807464
Berman, J. S., & Norton, N. C. (1985). Does professional training make a therapist more effective? Psychological Bulletin, 98, 401-407. DOI: 10.1037/0033-2909.98.2.401
Betan, E. J., & Binder, J. L. (2010). Clinical expertise in psychotherapy: How expert therapists use theory in generating case conceptualizations and interventions. Journal of Contemporary Psychotherapy,40, 141-152. DOI: 10.1007/s10879-010-9138-0
Beutler, L. E. (1997). The psychotherapist as a neglected variable in psychotherapy: An illustration by reference to the role of therapist experience and training. Clinical Psychology: Science and Practice, 4, 44-52. DOI: /10.1111/j.1468-2850.1997.tb00098.x
Beutler, L. E., Malik, M., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, S., & Wong, E. (2004). Therapist variables. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 227-306). New York, NY: Wiley & Sons, Inc.
Bohart, A. C., & Tallman, K. (2010). Clients: The neglected common factor in psychotherapy. In B. L. Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble. (Eds.), The heart and soul of change: Delivering what works in therapy (2nd ed., pp. 83-111). Washington, DC: American Psychological Association. DOI: 10.1037/12075-003
Budge, S. L., Owen, J. J., Kopta, S. M., Minami, T., Hansen, M. R., & Hirsch, G. (2013). Differences among trainees in client outcomes associated with the phase model of change. Psychotherapy, 50, 150-157. DOI: 10.1037/a0029565
Burlingame, G. M., & Barlow, S. H. (1996). Outcome and process differences between professional and nonprofessional therapists in time-limited group psychotherapy. International Journal of Group Psychotherapy, 46, 455- 478.
Castonguay, L. G., Goldfried, M. R., Wiser, S., Raue, P. J., & Hayes, A. M. (1996). Predicting the effect of cognitive therapy for depression: A study of unique and common factors. Journal of Consulting and Clinical Psychology, 64, 497-504. DOI: 10.1037/0022-006X.64.3.497
Chi, M. T. H. (2006). Two approaches to the study of experts’ characteristics. In Ericsson, K. A., Charness, N., Hoffman, R. R., & Feltovich, P. J. (Eds.), Cambridge handbook of expertise and expert performance (pp. 21-30). New York, NY: Cambridge University Press.
Chow, D., Miller, S. D., Seidel, J., Kane, R. T., & Andrews, X. (in press). The role of deliberate practice in the development of highly effective psychotherapy practice. Psychotherapy Research.
De Neys, W. (2012). Bias and conflict: A case for logical intuitions. Perspectives on Psychological Science, 7, 28–38. DOI: 10.1177/1745691611429354
Dawes, R. M. (1996). House of cards: Psychology and psychotherapy built on myth. New York, NY: Free Press.
Duncan, B. L., Miller, S. D., Wampold, B. E., & Hubble, M. A. (Eds.). (2010). The heart and soul of change: Delivering what works in therapy (2nd ed.). Washington, DC: American Psychological Association. DOI: 10.1037/12075-000
Eells, T. D., Lombert, K. G., Kendjelic, E. M., Turner, L. C., & Lucas, C. P. (2005). The quality of psychotherapy case formulations: A comparison of expert, experienced, and novice cognitive-behavioral and psychodynamic therapists. Journal of Consulting and Clinical Psychology, 73, 579–589. DOI: 10.1037/0022-006X.73.4.579
Ellis, S., Carette, B., Anseel, F., & Lievens, F. (2014). Systematic reflection: Implications for learning from failures and successes. Current Directions in Psychological Science, 23, 67–72. DOI: 10.1177/0963721413504106
Ericsson, K. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. J. Feltovich & Hoffman, R. R. (Eds.), The Cambridge handbook of expertise and expert performance (pp. 683–703). New York, NY: Cambridge University Press. DOI:10.1017/CBO9780511816796.038
Ericsson, K. A. (2009). Enhancing the development of professional performance: Implications from the study of deliberate practice. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 405–431). New York, NY: Cambridge University Press. DOI: 10.1017/CBO9780511609817.022
Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305. DOI: 10.1146/annurev.psych.47.1.273
Eugster, S. L., & Wampold, B. E. (1996). Systemic effects of participant role on evaluation of the psychotherapy session. Journal of Consulting and Clinical Psychology, 64, 1020-1028. DOI: 10.1037/0022-006X.64.5.102
Faust, D. (1991). What if we had really listened? Present reflections on altered pasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. 1. Matters of public interest (pp. 185-216). Minneapolis, MN: University of Minnesota Press.
Faust, D., & Ziskin, J. (1988). The expert witness in psychology and psychiatry. Science, 241(4861), 31-35. DOI: 10.1126/science.3291114
Faust, D. (2007). Decision research can increase the accuracy of clinical judgment and thereby improve patient care. In S.O. Lilienfeld & W.T. O’Donohue (Eds.), The great ideas of clinical science: 17 principles that every mental health professional should understand (pp. 49-76). New York, NY: Routledge.
Garb, H. N. (1989). Clinical judgment, clinical training, and professional experience. Psychological Bulletin, 105, 387-396. DOI: 10.1037/0033-2909.105.3.387
Garb, H. N. (1997). Race bias, social class bias, and gender bias in clinical judgment. Clinical Psychology: Science and Practice, 4, 99–120.
Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association. DOI: 10.1037/10299-000
Garb, H. N. (2005). Clinical judgment and decision making. Annual Review of Clinical Psychology, 1, 67–89. DOI: 10.1146/annurev.clinpsy.1.102803.143810
Goldberg, S., Rousmaniere, T. G., Miller, S. D., Whipple, J. L., Nielsen, S. L., & Wampold, B. E. (submitted). Do psychotherapists improve with time and experience? A longitudinal analysis of real world outcome data.
Groth‐Marnat, G., Roberts, R. I., & Beutler, L. E. (2001). Client characteristics and psychotherapy: Perspectives, support, interactions, and implications for training. Australian Psychologist, 36, 115-121. DOI: 10.1080/00050060108259643
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30. DOI: 10.1037/1040-3522.214.171.124
Hattie, J. A., Sharpley, C. F., & Rogers, H. F. (1984). Comparative effectiveness of professional and paraprofessional helpers. Psychological Bulletin, 95, 534-541. DOI: 10.1037/0033-2909.95.3.534
Hendlin, S. J. (2014). Finding expertise in the process. Psychotherapy Bulletin, 49(4), 9-11.
Henry, W. P., Strupp, H. H., Butler, S. F., Schacht, T. E., & Binder, J. L. (1993). Effects of training in time-limited psychotherapy: Changes in therapist behavior. Journal of Consulting and Clinical Psychology, 61, 434-440. DOI: 10.1037/0022-006X.61.3.434
Huppert, J. D., Bufka, L. F., Barlow, D. H., Gorman, J. M., Shear, M. K., Woods, S. W. (2001). Therapists, therapist variables, and cognitive-behavioral therapy outcome in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69(5), 747-755. DOI: 10.1037/0022-006X.69.5.747
Jennings, L., & Skovholt, T. M. (1999). The cognitive, emotional, and relational characteristics of master therapists. Journal of Counseling Psychology, 46, 3-11. DOI: 10.1037/0022-0126.96.36.199
Kivlighan, D. M., & Quigley, S. T. (1991). Dimensions used by experienced and novice group therapists to conceptualize group process. Journal of Counseling Psychology, 38, 415-423. DOI: 10.1037/0022-0188.8.131.525
Kaslow, N. J. (2004). Competencies in professional psychology. American Psychologist, 59, 774-781. DOI: 10.1037/0003-066X.59.8.774
Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61, 165-174. DOI: 10.1002/jclp.20113
Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48(1), 72-79. DOI: 10.1037/a0022238
Laska, K. M., Smith, T. L., Wislocki, A. P., Minami, T., & Wampold, B. E. (2013). Uniformity of evidence-based treatments in practice? Therapist effects in the delivery of cognitive processing therapy for PTSD. Journal of Counseling Psychology, 60, 31-41. DOI: 10.1037/a0031294
Linehan, M. M., Comtois, K. A., Murray, A. M., Brown, M. Z., Gallop, R. J., Heard, H. L., … & Lindenboim, N. (2006). Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Archives of General Psychiatry, 63, 757–766. DOI: 10.1001/archpsyc.63.7.757
Martin, J., Slemon, A. G., Hiebert, B., Hallberg, E. T., & Cummings, A. L. (1989). Conceptualizations of novice and experienced counselors. Journal of Counseling Psychology, 36, 395-400.
McMahan, E. H. (2014). Supervision, a nonelusive component of deliberate practice toward expertise. American Psychologist, 69, 712-713. DOI: /10.1037/a0037832
Meehl, P. E. (1954). Clinical vs. statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis, MN: University of Minnesota Press.
Meehl, P. E. (1986). Causes and effects of my disturbing little book. Journal of Personality Assessment. 50, 370-375. DOI: 10.1207/s15327752jpa5003_6
Miller, S. D., Duncan, B. L., & Hubble, M. A. (2005). Outcome-informed clinical work. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration (2nd ed., pp. 84-102). New York, NY: Oxford University Press.
Miller, S. D., Duncan, B. M., Sorrell, R., & Brown, G. S. (2005). The partners for change outcome management system. Journal of Clinical Psychology, 61(2), 199–208.
Miller, S.D., Hubble, M. A., Chow, D, L. & Seidel, J. A. (2013). The outcome of psychotherapy: Yesterday, today, and tomorrow. Psychotherapy, 50, 88–97. DOI: 10.1037/a0031097
Miller, S. D., Hubble, M., & Duncan, B. (2008). Supershrinks: What is the secret of their success. Psychotherapy in Australia, 14, 14-22.
Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E. G., Brown, G. S., & Kircher, J. (2008). Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment: A preliminary study. Journal of Consulting and Clinical Psychology, 76, 116-124. DOI: 10.1037/0022-006X.76.1.116
Oddli, H. W., Halvorsen, M. S., & Rønnestad, M. H. (2014, November). Expertise demonstrated: What does it mean to be an expert psychotherapist? Retrieved from http://societyforpsychotherapy.org/expertise-demonstrated
Overholser, J. C. (2010). Clinical expertise: A preliminary attempt to clarify its core elements. Journal of Contemporary Psychotherapy, 40, 131–139. DOI 10.1007/s10879-009-9129-1
Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003). In search of supershrink: Using patient outcome to identify effective and ineffective therapists. Clinical Psychology and Psychotherapy, 10, 361-373
Rector, N. A., & Cassin, S. E. (2010). Clinical expertise in cognitive behavioural therapy: Definition and pathways to acquisition. Journal of Contemporary Psychotherapy, 40, 153-161. DOI: 10.1007/s10879-010-9136-2
Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training, 46, 418-431. DOI: 10.1037/a0017901
Rønnestad, M. H., & Skovholt, T. M. (2013). The developing practitioner. Growth and stagnation of therapists and counselors. New York, NY: Routledge/Taylor & Francis Group.
Rousmaniere, T. G., Swift, J. K., Babins-Wagner, R., Whipple, J. L. & Berzins, S. (2014). Supervisor variance in psychotherapy outcome in routine practice. Psychotherapy Research, 2, 1-10. DOI: 10.1080/10503307.2014.963730
Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252-266. DOI: 10.1037/a0037874
Shanteau, J., & Weiss, D. J. (2014). Individual expertise versus domain expertise. American Psychologist, 69, 711-712. DOI: 10.1037/a0037874
Spengler, P. M., White, M. J., Ægisdóttir, S., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G. R., & Rush, J. D. (2009). The meta-analysis of clinical judgment project: Effects of experience on judgment accuracy. The Counseling Psychologist, 37, 350-399. DOI: 10.1177/0011000006295149
Stein, D, M., & Lambert, M. J. (1984). On the relationship between therapist experience and psychotherapy outcome. Clinical Psychology Review, 4, 127-142. DOI: 10.1016/0272-7358(84)90025-4
Stein, D, M., & Lambert, M. J. (1995). Graduate training in psychotherapy: Are therapy outcomes enhanced? Journal of Consulting and Clinical Psychology, 63, 182-196. DOI: 10.1037/0022-006X.63.5.717
Strupp, H. H., & Hadley, S. W. (1979). Specific vs nonspecific factors in psychotherapy: A controlled study of outcome. Archives of General Psychiatry, 36, 1125-1136. DOI: 10.1001/archpsyc.1979.01780100095009
Skovholt, T. M., & Jennings, L. (Eds.). (2004 ). Master therapists: Exploring expertise in therapy and counseling. Boston, MA: Allyn & Bacon.
Tracey, T. J. G., Lichtenberg, J. W., Goodyear, R. K., Claiborn, C. D., & Wampold, B. E. (2003). Concept mapping of therapeutic common factors. Psychotherapy Research, 13, 401-413. DOI:10.1093/ptr/kpg041
Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69, 218–229. DOI: 10.1037/a003509
Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620-630. DOI: 10.1037/0022-006X.61.4.620
Wampold, B. E., & Brown, G. S. (2005). Estimating variability in outcomes attributable to therapists: A naturalistic study of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73, 914-923. DOI: 10.1037/0022-006X.73.5.914
Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: Research evidence for what works in psychotherapy (2nd ed.). New York: Routledge.
Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200-211. DOI: 10.1037/a0018912
Wierzbicki, M. (1993). Issues in clinical psychology: Subjective versus objective approaches. Boston, MA: Allyn & Bacon.
Witteman, C. L. M., Weiss, D. J., & Metzmacher, M. (2012). Assessing diagnostic expertise of counselors using the Cochran-Weiss-Shanteau (CWS) index. Journal of Counseling and Development, 90, 30-34. DOI: 10.1111/j.1556-6676.2012.00005.x