Here’s a quick tip: when a study that purports to find evidence of treatment effectiveness–preliminary or not–doesn’t have a control group (a group that doesn’t undergo treatment but is otherwise similar to the group that does), you should raise your eyebrows. Or shake your head. Or roll your eyes. Whichever you prefer.
Why do we need a control group? If the treatment works, we will see improvement in the patients, so isn’t that evidence enough? Well, no.
[T]he whole idea of an experiment is to identify two identical groups of people and then to manipulate something. One group gets an experimental treatment, and one does not. If the group that gets the treatment (e.g., a drug, exposure to a violent video game) behaves differently than the control group that did not get the treatment, we can attribute the difference to the treatment – but only if we can rest assured that the two groups were similar prior to the treatment…
The key issue is that it is always important to have a control group if you want to assess the impact of a treatment.
(By the way, I recommend reading the article I just quoted, “How to Be a Wise Consumer of Psychological Research” from the American Psychological Association.)
In the introduction, the authors of this study make the case that anorexia nervosa (AN) is difficult to treat and difficult to study (low prevalence, high dropout, necessary long-length of follow-up, etc…) and so if we could have treatment approach that would be successful on an outpatient basis, that would be great.
No argument there. There is a lot of consensus that long-term continuous low-intensity care is crucial for sustained recovery. (A short stay at a hospital might get the patient out of the medical danger zone, but it is highly unlikely to cure them.)
CBT, or cognitive-behavioural therapy, is something that could be administered on an outpatient basis, and the authors claim, “is the leading empirically supported treatment for bulimia nervosa (NICE, 2004; Shapiro et al., 2007).” (I’ve got to check those sources myself, before I can feel comfortable claiming that.)
And by the way, I want to be especially clear in posts that I write about treatment: I am interested in evidence. There are a lot of recovery-oriented blogs out there, and plenty of blogs supporting one treatment modality over another. This isn’t one of them. I’m simply interested in evidence, good research and good science (I have no monetary or psychological stake in this.)
Back to the study, the authors wanted to evaluate whether an “enhanced” form of CBT (which “focuses on modifying the mechanisms thought to perpetuate all forms of eating disorder psychopathology”) is effective in treating adult anorexia nervosa patients.
CBT-E is a treatment for patients with eating disorder psychopathology. With patients who are underweight, it has three phases. In the ﬁrst, the emphasis is on increasing patients’ motivation to change. Then, if willing, patients are helped to regain weight while at the same time they tackle their eating disorder psychopathology including their extreme concerns about shape and weight. In the ﬁnal phase the emphasis is on helping them develop personalized strategies for identifying and immediately correcting any setbacks.
Patient recruitment was done in the UK and in Italy. All of the patients partook in 40 one-on-one CBT-E sessions that lasted 50 minutes each, over a 40 week period. None of the participants were involved in any other kind of psychological treatment. They were followed up for 60 weeks after treatment CBT-E sessions ended.
Patients fulfilled the DSM-IV criteria for AN (amenorrhea was not a criteria) and were between BMI 15-17.5 (Italy cohort didn’t have a minimum BMI threshold). All were between 18-65 years of age. The majority were single, female, and in their mid-20’s. They had an ED for an average of 3 years. Patients could not have had specialist treatment for AN within the previous year and they could not have a coexisting Axis 1 psychiatric disorder. Thirty-nine (out of ninety-nine) patients were on “clinically warranted” (what does that mean?) antidepressant medication and were kept stable on the medication for the duration of the CBT-E.
So what did they find?
- ~ 1/3 of the initial 99 participants did not complete the study
63 patients completed CBT-E, 53 of those were assessed 60-weeks post-treatment:
- mean BMI before treatment was 16.0 (standard deviation, SD 1.2), after treatment: 18.8, and 60-weeks post-treatment: 18.7 (SD 2)
- 39/63 had a BMI > 18.5 right after treatment, and 60-weeks after that number decreased to 30/54.
- before treatment: 27.4%, 20.6% and 7.9% of the sample engaged in binge eating, self-induced vomiting, and laxative misuse, respectively (out of 63 patients)
- 60-weeks post-treatment: 18.5%, 13% and 9.3% of the sample engaged in binge eating, self-induced vomiting and laxative misuse, respectively (out of 53 patients)
- in terms of ED psychopathology (measured by EDE-Q), scores for overall severity decreased across the board following treatment and then increased during the 60-week follow-up assessment, on average of about 85th percentile before treatment to about 50th percentile 60-weeks after treatment (Luce et al., 2008)
Out of the 63 patients that completed treatment, 5 received “significant additional treatment” (the authors did not mention what that entailed) and 7 others received five CBT-E “booster” sessions.
What were the main differences between patients that completed CBT-E from those that did not?
As you might guess, the more severe the eating disorder and general psychopathology, the less likely the individuals were to complete the treatment. Binge eating frequency, purging frequency, and EDE-Q scores were higher in those that did not complete the treatment. Isn’t CBT touted as the most “evidence-based” method for bulimia? It is the “leading empirically supported treatment for bulimia nervosa,” according to the authors. You would think that anorexia nervosa patients that are bingeing and purging would be more likely to reap the benefits of CBT, at least in terms of decreasing the frequency of bingeing and purging, if nothing else. This seems somewhat odd to me.
I mean, I get it, they were sicker overall, but still, I would expect patients that were bingeing and purging the most to see the biggest improvements in that regard–maybe not overall EDE-Q psychopathology but at least for the symptoms that Fairburn claims have the best outcomes with CBT.
Why did patients drop-out of treatment?
Two main reasons: participants either just stopped coming or they were withdrawn because of their physical health or lack of progress.
This is a great way to positively skew the data: you enroll 100 patients, 35 drop because it is really not working for them, and from the remaining 65, half end up improving. When you are counting just the ones that completed the treatment (meaning they were making some progress) about half seemed to benefit. But looking at the whole picture, only a third seemingly experienced benefits from the treatment.
What about maintaining change after treatment?
The third ﬁnding is therefore of particular interest. It concerns the stability of the changes obtained. Despite there being little exposure to further treatment, the changes were generally well maintained with there being only a slight deterioration in weight and eating disorder features. This is in marked contrast to the reports of high rates of relapse over the 12 months following hospitalization, even with ongoing therapeutic input (Carter et al., 2009; Kaplan et al., 2009; Walsh et al., 2006)
Of course you’d expect to see that when you only keep the most successful patients, right?
Remember: 99 of patients enrolled, 63 finished CBT-E and 53 completed follow-up. So, in the 53 that completed follow-up there was “minimal” deterioration. But what about the ones that didn’t complete the study? What about the 15% that completed CBT-E but didn’t follow-up? If there was deterioration for notable deterioration in the patients that came for follow-up, you can image the deterioration for those that didn’t could be even worse. Maybe much worse.
After all, if you partook in treatment, and did okay-ish, and then a year later you felt you were back to square one, would you want to go and admit that to the doctors that spent their time treating you and were proud of your original progress? Maybe, but probably not.And here is another point that’s conveniently glossed over in the discussion: antidepressant medication.
Thirty-nine out of the original ninety-nine patients were on antidepressant medication prior to starting, and following the completion of the study. There was NO mention of the medication status of patients that completed the study. Fifty-three completed follow-up, and as far as I’m concerned, not having this information available, thirty-nine of them could’ve been on antidepressant medication–which they started PRIOR to enrolling in this study.
Here’s a scenario: 39 patients start antidepressants prior to the study, and for some of them, this will decrease their depression and/or anxiety. Say they start this a month or so prior to the study, so when the study starts, they are just beginning to reap the benefits of the medication. Say for those patients that feel less depression and less anxious, they might think less about food or weight, maybe they’ll even start eating a bit more or reduce the time they spend exercising.
And so they begin to get a little better. Antidepressants will not cure anorexia nervosa, but they might decrease anxiety enough to enable some patients to more easily put on a few pounds and maybe even feel more motivated to eat. Of course, this will not be the case for most, or even the majority of those that go on antidepressants, but it will probably be the case for some.
Now given that it is not stated which of these 39 patients completed the study and which dropped (major omission that I think is ridiculous), how can we make any conclusions about the effect of CBT? Out of the original 99, 60-weeks following treatment for the 53 that completed that part of the study, only 30 had a BMI above 18.5, and the average BMI was 18.7.
We simply cannot determine if the positive effects we are seeing here are due to CBT-E, antidepressant medication OR other factors (placebo: regression to the mean, not-treatment related improvement, etc..).
There were more limitations, and although Fairburn et al mentioned them first, I don’t think they were the biggest problems in this study: 1) we don’t know if the findings here can be generalized to those below BMI of 15.0 or above 17.5; 2) we need longer follow-up; 3) this study had only “modest statistical power” for detecting site effects. What does that mean? It means, in short: we need a lot more people if we really want to determine that this treatment works or doesn’t work.
Anyway, I’m not saying that CBT-E is or isn’t effective for the treatment of adult patients with anorexia nervosa, what I’m saying is that this is not something we can determine from this study–at all–because there was no control group. And if you want to to know whether your treatment or intervention had any positive or negative effects, you absolutely have to have a control group.
EDIT: Missie Plotkin rightfully pointed out on the Facebook page that I didn’t emphasize that this was a preliminary study. I mentioned in my first sentence that regardless of whether a study is preliminary or not, controls are important, but forgot the mention this study was meant to be a very preliminary study to provide evidence for larger randomized controlled trials. Nonetheless, any study that is meant to show that a treatment has a positive effect has to have controls for us to draw any conclusions. Otherwise, it is nearly impossible to tell whether the results are due to the treatment or to other factors. So, what’s the point of a prelim study where we can’t tell if the effect is due to the treatment or not, frankly, I’m not sure.