Valuing Complex Data without Devaluing the P-Value
Internet Editor’s Note: Dr. Tanofsky-Kraff and her colleagues recently published an article titled “Interpersonal psychotherapy for the prevention of excess weight gain and eating disorders: A brief case study” in Psychotherapy.
If you’re a member of the Society for the Advancement of Psychotherapy you can access the Psychotherapy article via your APA member page.
Not a member? Purchase the Psychotherapy article for $11.95 here.
Or, Join the Society for $40 a year and receive access to more than 50 years of articles.
Since 1992, I have been exposed to psychotherapy research, either working on other researchers’ trials or as a principal investigator. Of the time-limited approaches to which I have been exposed, interpersonal psychotherapy (IPT) resonates with me as a therapist, a clinical supervisor, and a mentor.
I have observed IPT meaningfully impact the lives of clients and study participants – young and old. Relatedly, my career has been defined by a strong commitment to applying rigorous scientific methods. This commitment is combined with a healthy appreciation that the complexity and depth of the human experience is not easily captured by measures and metrics. To advance the science of psychotherapy, I believe we need methods that are rigorous, but not so rigid that we miss the nuances and hypothesis generating value of post hoc or secondary analyses.
My experience of IPT as a life changing therapeutic approach drives my interest to continue studying its mechanism even when the findings are not precisely as anticipated. Despite being grounded in theory (Weissman, Markowitz, & Klerman, 2000) and prior data (Tanofsky-Kraff et al., 2010), our group found few statistically significant differences in preventing excess weight gain between an adapted IPT (Tanofsky-Kraff, Shomaker, Young, & Wilfley, 2016) and a standard-of-care health education comparison group for adolescent girls at high-risk for obesity after one year (Tanofsky-Kraff et al., 2014).
However, by three years, youth with more anxiety and social problems experienced the greatest age-adjusted BMI loss and adiposity stabilization if they were randomized to IPT (Tanofsky-Kraff et al., in press). These findings are consistent with interpersonal theory (Weissman et al., 2000), and remarkably parallel to other IPT research.
In adults with eating disorders (Wilson, Wilfley, Agras, & Bryson, 2010), and adolescents with depression (Young, Gallop, & Mufson, 2009; Young, Mufson, & Davies, 2006; Gunlicks-Stoessel, Mufson, Jekal, & Turner, 2010), psychological functioning at presentation appears to moderate outcome such that those with more problems are highly responsive to IPT.
These data support a compensation model of psychotherapy (Rude & Rehm, 1991), that individuals may be especially responsive to interventions that target their underlying problems, and speak to the utility of IPT for high-risk and clinical populations.
Implications of These Findings
I consider these findings important, particularly given that overweight-related concerns and obesity are difficult to impact among adolescents. As with all studies, replication is required. Yet, for a subset of youth who were willing to participate in our trial, we are hopeful that we may have altered a trajectory of continued excess weight gain and possibly prevented the development of physiological and psychological problems associated with obesity.
Among psychotherapists, there is a sense of optimism that we may potentially reduce the high rates of obesity for several years in a hard-to-treat group using an approach that is non-invasive and promotes positive relationships and psychological functioning.
Detractors point out that our primary hypothesis did not reach the p-value cut-off of .05 and moderator analyses are post-hoc and only impact a subset. These concerns are warranted as a means to advance rigorous science.
However, in pursuit of rigorous science, we should not lose the opportunity to learn from the entirety of our data (what worked for some subgroups and why) even when the primary outcome does not meet p ≤ .05. These are crucial data to inform future psychotherapy research and inform clinicians about potentially promising nuances of treatment delivery.
The Importance of Sub-Groups, Over and Above the P-Value
Unlike providing a medication or surgery (which include their own variability), psychotherapy is harder to control scientifically. Further complicating control are the interactions among therapist, psychotherapeutic approach, and individual psychological differences of clients.
As a result, our field must not be dismayed when we identify subgroups to better match treatment to person. This is not a novel concept. Since 1967, there have been calls for elucidating targeted therapies (“What treatment, delivered by whom, is most effective for what problem, under which set of circumstances”) (Paul, 1967).
Indeed, mice are not men (or women!). Despite the broad heterogeneity and nuances of human behavior, when taken to an extreme, applying rigid scientific hypothesis testing and interpretation to psychotherapy research may be analogous to putting a square peg into a circular hole. Indeed, even our best therapies for broad psychological disorders only impact approximately 50% of patients (Insel et al., 2010).
Why is this?
At the most basic level, people are heterogeneous and change happens based on one’s internal and external environment with far greater impact on psychotherapeutic approaches as compared to physical interventions. If a behavior or outcome is observed that may not have been the initial hypothesis, but the methodological rigor of the study was intact, the scientific community should be open to the unexpected.
Indeed, theories are just that – theories! There should be no disgrace in not supporting a theory or hypothesis. Yet, all researchers regardless of discipline, are often shamed and this results in actions that hinder science; for example the “file drawer” or “be first or be best” publication syndromes. This is especially problematic because understanding why a study replicates – or does not replicate – is central to science.
Conclusion and Summary
At times, we are fortunate to work with savvy collaborators and editors who can see the value of clear scientific rigor, alongside of the robust variability of our human participants and their environment. Recently, I was delighted to read an article in the New England Journal of Medicine that discussed a more optimistic approach to clinical trials; namely, interpreting results in the context of several aspects and not deeming a trial as a failure if the primary outcome does not reach the arbitrary p-value cut-off (Pocock & Stone, 2016).
Clearly, boundaries are important and I am by no means a supporter of “fishing expeditions”. However, we must enlighten scientists to restrain from “throwing out the baby with the bath water” when interpreting trial outcomes, particularly those involving psychotherapy. Watering down or ignoring our results and interpretations will only result in a disservice to the field, and more disturbingly, to clients with psychological difficulties who deserve to have every bit of science considered when seeking treatment.
Cite This Article
Tanofsky-Kraff, M. (2016, October). Psychotherapy science: Valuing complex data without devaluing the p-value. [Web article]. Retrieved from https://societyforpsychotherapy.org/psychotherapy-science-valuing-complex-data
Field, A. E., Camargo, C. A., & Ogino, S. (2013). The merits of subtyping obesity: one size does not fit all. JAMA: The Journal of the American Medical Association, 310(20), 2147-2148.
Gunlicks-Stoessel, M., Mufson, L., Jekal, A.,& Turner, J. B. (2010). The impact of perceived interpersonal functioning on treatment for adolescent depression: IPT-A versus treatment as usual in school-based health clinics. Journal of Consultation and Clinical Psychology, 78(2), 260-267.
Insel, T., Cuthbert, B. N., Garvey, M. A., Heinssen, R., Pine, D. S., Quinn, K. …Wang, P. (2010). Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. American Journal of Psychiatry, 167(7), 748-751.
Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31(2), 109-118.
Pocock, S. J., & Stone, G. W. (2016). The primary outcome fails – What next?New England Journal of Medicine, 375(9), 861-870.
Rude, S. S., & Rehm, L. P. (1991). Response to treatments for depression: The role of initial status on targeted cognitive and behavioral skills. Clinical Psychology Review, 11, 493-514.
Tanofsky-Kraff, M., Shomaker, L. B., Wilfley, D. E., Young, J. F., Sbrocco, T., Stephens, M., …Yanovski, J. A. (2014). Targeted prevention of excess weight gain and eating disorders in high-risk adolescent girls: a randomized controlled trial. The American Journal of Clinical Nutrition, 100(4), 1010-1018.
Tanofsky-Kraff, M., Shomaker, L. B., & Wilfley, D. E. (in press). Excess weight gain prevention in adolescents: Three-year outcome following a randomized-controlled trial. Journal of Consulting and Clinical Psychology.
Tanofsky-Kraff, M., Shomaker, L. B., Young, J. F., & Wilfley, D.E. (2016). Interpersonal psychotherapy for the prevention of excess weight gain and eating disorders: A brief case study. Psychotherapy, 53(2), 188-194.
Tanofsky-Kraff, M., Wilfley, D. E., Young, J. F., Mufson, L., Yanovski, S. Z., Glasofer, D. R., … Schvey, N. A. (2010). A pilot study of interpersonal psychotherapy for preventing excess weight gain in adolescent girls at-risk for obesity. International Journal of Eating Disorders, 43(8), 701-706.
Weissman, M. M., Markowitz, J., & Klerman, G. L. (2000). Comprehensive guide to Interpersonal psychotherapy. New York: Basic Behavioral Science Books.
Wilson, G. T., Wilfley, D. E., Agras, W. S., & Bryson, S. W. (2010). Psychological treatments of binge eating disorder. Archives of general psychiatry, 67(1), 94-101.
Young, J. F., Gallop, R., & Mufson, L. (2009). Mother-child conflict and its moderating effects on depression outcomes in a preventive intervention for adolescent depression. Journal of Clinical Child & Adolescent Psychology,38(5), 696-704.
Young, J. F., Mufson, L., & Davies, M. (2006). Impact of comorbid anxiety in an effectiveness study of interpersonal psychotherapy for depressed adolescents.Journal of the American Academy of Child & Adolescent Psychiatry, 45(8), 904-912.