This is Part II of my mini-series on the Mandometer(r) treatment for eating disorders (link to Part I). In Part I, I provided some background on the Mandometer(r) treatment; in this post, I want to take an in-depth look at the recent Mandometer treatment study. My main goal is to see whether their data live up to their claims. Warning: This post may contain high levels of snark.
Their main claims? This is from the abstract:
The estimated rate of remission for this therapy was 75% after a median of 12.5 months of treatment. A competing event such as the termination of insurance coverage, or failure of the treatment, interfered with outcomes in 16% of the patients, and the other patients remained in treatment. Of those who went in remission, the estimated rate of relapse was 10% over 5 years of follow-up and there was no mortality.
Sounds pretty good, right? (Note the use of the word “estimated.“)
From 1993 to 2011, Bergh et al. followed 1,428 consecutively admitted patients to the six clinics participating in this study (3 in Sweden, 1 each in Amsterdam, Melbourne, and San Diego). 1,428 is an impressive number for an eating disorders treatment study and the multi-site nature of the study is also a positive. In the last 3 years, individuals were able to self-refer themselves; up to 70% of patients now enter the treatment through self-referral.
In addition to completing several questionnaires (Eating Disorder Inventory-2, Comprehensive Psychopathological Self-Rating Scale, and a quality of life questionnaire), patients had to eat a meal without feedback to determine “speed of eating, the amount of food eaten, and their development of satiety over the course of the meal.”
Out of the 1,428 patients, 251 were classified as severely ill (e.g., BMI <13.5, low body temperature) and were initially treated as inpatients. Overall, 40% of the patients had anorexia nervosa (AN), 17% had bulimia nervosa (BN), and 43% had eating disorder not otherwise specified (EDNOS).
The average age was 17.5, 22.6, and 20.5, for AN, BN, and EDNOS patients, respectively. Average BMIs were 14.9, 21.5, and 18.5, for AN, BN, and EDNOS patients, respectively. AN patients had the shortest average illness duration (3.2 years) and BN patients the longest (7.4 years).
As mentioned in Part I, the Mandometer treatment is focused on normalising eating speed and volume, providing external heat, and minimizing physical activity. Bergh et al.:
Briefly, the patients normalize their eating pattern with mealtime feedback provided by a scale that rests under a dinner plate, connected to a small computer. By consulting a small monitor next to their plate, patients are able to compare their rate of eating in real time to that of a typical person eating that meal. The patients also develop normal feelings of satiety using the same strategy.
Initially, a behavioral therapist assists the patients, but the patients get used to the procedure rapidly and can then practice eating without the support of a therapist, including practicing at home. In addition, the patients are provided with warmth, using warm rooms, thermal blankets, or jackets to calm them and to avoid the use of calories for thermoregulation.
Their physical activity is restricted for that same purpose, and great deal of time is spent convincing and coaxing the patients to start resuming their normal social interactions. Approximately 30% of the patients were taking psychoactive drugs on admission and these are gradually withdrawn over the first months of treatment.
That’s it! It seems so easy! Why isn’t everyone on board? Why isn’t everyone as excited as this press release suggests we should be? (My favourite part of the press release: Someone apparently suggested that the Mandometer treatment is on par with the discovery of penicillin).
The study’s definition of remission was quite comprehensive and holistic (a good thing, of course): Patients were considered to be in remission when “they no longer meet the criteria for an eating disorder, when their body weight, eating behavior, feelings of satiety, physiological status, level of depression, anxiety, and obsession are normal, when they are able to state that food and body weight are no longer a problem, and when they are back at school or work. Bulimic patients must in addition have stopped bingeing and purging for at least 3 months.”
In addition, “meeting five of these criteria was regarded as ‘partial remission’ starting in 2009.” Since it doesn’t seem to matter which criteria you fulfil, you can technically be in partial remission but meet the criteria for an eating disorder (for example: have a normal body weight, normal physiological status, normal depression, anxiety, and obsessionality scores, be back to work, and state that food and weight aren’t a problem).
Now the fun part: The Results
Bergh at al. dedicate three figures and one data table for the results.
The first figure shows the factors affecting the rates of remission, though it tells us nothing interesting: overall, a higher admission BMI, a diagnosis of EDNOS, and decreased illness severity predicted a faster rate of remission (Duh). What you can see in the graphs below is that as time goes on a larger proportion of patients remit. In D, for example, you can see that by 48 month for example, a bigger proportion of EDNOS patients than BN or AN patients had remitted:
I highlighted some interesting parts.
1. The authors did NOT define “ill” anywhere in the paper; they defined “severely ill,” but not “ill”. (All of the patients were diagnosed with an ED, so technically, weren’t they all ill?)
2. With the exception of the figure caption, there is NO mention whatsoever of the number of patients that fulfilled 5/7 of the seven criteria for remission. Nowhere — not even in the results table.
What’s more, the table below and everything else on remission only cites the 737 (“full remission”) number. From the wording, it sounds like an additional 378 fulfilled partial remission, but this doesn’t make any sense with respect to any of the other data the authors cite: Adding the numbers of fully remitted patients, those who were censored, withdrew or “failed” = 1,428. Where are these partial remits?
3. What seven criteria? I tried to find 7 from the blurb I quoted above on what qualifies as remission and I couldn’t count seven. Moreover, the individuals in the BN category have an additional criterion (abstaining from bingeing/purging for 3 months), is that included in the 7 (and thus AN patients only have six criteria to fill?) or it is a extra one (and thus BN patients have 8 criteria to fill)? These may seem like minor points, but this lack of clarity is unacceptable for a peer-reviewed paper.
4. This is also the first time I’ve ever seen authors omit important information (confidence intervals) to “facilitate visual inspection” without including those numbers anywhere in the paper. With a few exceptions, confidence intervals, p-values, and effect sizes are conspicuously missing from the entire paper.
But moving on. Bergh et al. have a figure illustrating the effect of a “competing event” (in other words: insurance cut out) on treatment in the San Diego clinic. The final figure shows the proportion of patients who have relapsed over 5 years:
The study recruited patients up to 2011, so I wonder how they got 5-year follow-up data from those admitted in 2010 and who, say, remitted in 2011? Are these researchers psychic?
And now, the grand finale. The final and only table that shows treatment outcomes:
The second column shows the number of patients in each clinic. The remission column shows the number of remitted patients. Note that there’s no information on partial versus full remission. Where do those 378 partially remitted patients fit?
Data from patients who were still in treatment or who dropped out for unknown reasons were censored. The effect of a “competing event,” which interfered with the possibility of going into remission, was analyzed using cause-specific hazard functions and corresponding cumulative incidence estimators (Gray, 2011). Competing events included instances when patients withdrew because they felt that they were not improving, when they were diagnosed with an unrelated illness, or when they withdrew because of financial constraints. That is to say, incidents that were either related (treatment failure) or unrelated (e.g., insurers did not pay) to the therapy could prevent the patient from completing the treatment and were therefore considered competing events.
This part of the paper really infuriated me:
Patients withdrawn for reasons unrelated to the treatment were censored at the time when they were no longer in treatment. Failure of insurers to pay for treatment in San Diego provided an example and explained why 50% of the patients were censored in this clinic (see Table 3). Patients withdrawn for reasons related to the treatment, that is, the treatment failed, were retained in the analysis, “burdening” the denominator in the calculation at all times, thus yielding a conservative estimate of the rate of remission.
First of all, Bergh et al. keeping pointing to insurance as a reason for “censorship” but this only really applies to the US clinic (which made up a paltry 4% of the total sample). What about the other places?
But more importantly, keeping the patients who withdrew because treatment failed or for reasons related to treatment does NOT yield a conservative measure! How else would you calculate the percentage of patients who have remitted if not by looking at the proportion of those who remitted out of the total number of patients who entered the study, which includes those who haven’t remitted because treatment did not work? Including those who haven’t remitted because “treatment failed” does NOT IN ANY WAY make the measure “conservative.”
Their way of calculating the probability of going into remission (“life tables” and survival analyse) yielded interesting numbers:
In Amsterdam, where 98 patients entered treatment, 24 remitted. Bergh et al. calculated that the probability of going into remission in this clinic was 64%. In Danderyd, 205 patients entered treatment and 141 remitted. Bergh et al. calculated that there’s was an 86.4% chance of going into remission. In Huddinge, 918 patients entered treatment, 490 remitted. The probability of going into remission in this clinic was, according to Bergh et al., 74%.
Now, I’m not a statistician. Not even close. But it seems disingenuous to calculate remission rates this way. After all: 24/98 = 24.5% (not 64%); 141/205 = 68.7% (not 86%); 490/518 = 53.4% (not 74%). This is how they get an “estimated” 75% remission rate when only 51.5% of the initial sample remitted (737/1428 = 51.6%).
This study also did not have a control group — so we can’t compare the Mandometer Treatment to another treatment protocol or even a waitlist control. How many of these patients would recover without the Mandometer? We don’t know.)
I’ve never seen life tables used in an eating disorders treatment study. Now that might be because I haven’t read enough studies, but you do have to wonder why, in 2013, Bergh et al. would chose to use a non-standard way to analyse a treatment study of this nature? Survival analyses have many uses, but as far as I’m aware, this isn’t one of them. (Please correct me if I’m wrong, though.)
[EDIT: A friend told me of a treatment study from a group in Toronto that used survival analyses to evaluate time to remission (or relapse? can’t recall). She will link me to the study so I can compare the calculations. So, correction: they are used, but not that frequently.]
Bergh et al. also didn’t use the standard/most commonly used questionnaires to evaluate anxiety, depression, and obsessionality. Instead, they used a questionnaire that another group found underestimated the levels of anxiety, depression and obsessionality.
AND AS IF THIS ISN’T ENOUGH…
This might be the FIRST treatment study I have ever seen that had no information whatsoever (anywhere!) on the remitted patients (with the exception of the “estimated” 10% relapse rate, though they just show it in the figure and provide NO follow-up date).
Bergh et al. provide NO information on:
- The extent of weight restoration in remitted patients (no BMI values or % of expected body weight or any kind of metric that enables readers to compare the participants’ weight before and after treatment
- The levels of anxiety, depression, and obsessionality in remitted patients
- Eating Disorder Psychopoathology (EDI-2) and Quality of Life measures (actually, these were NOT reported before or after treatment, though the authors said they collected the data)
- Menstrual status
- Symptom frequency
- The proportion of individuals who have jobs or go to school (recall this was one of the seven criteria for remission)
- Comorbid disorders anywhere — before or after — and you’d think for researchers who claim that comorbid psychopathology is the result of malnutrition and exercise, they’d want to provide data to show how comorbid psychopathology decreased following treatment
LET’S RECAP, SHALL WE?
Bergh et al.:
- Did not have a control group
- Did not have clear remission criteria
- Did not use standard approaches to calculate remission
- Did not report post-treatment data for assessments of anxiety, depression, and obsessionality (except for the 13 patients that dropped out of the San Diego clinic)
- Did not report eating disorder pathology, menstrual status, or symptom frequency before or after treatment
- Did not report any information on psychiatric comorbidities
- Did not provide information on any remission criteria in patients who remitted (extent of weight restoration being a crucial one), with the exception of relapse rates
- Did not provide clear information about the partially remitted patients (only mentioned the number in a figure caption) and it is unclear where those patients fit (as the numbers don’t add up)
- Did not include confidence intervals for the vast majority of the data (not even in text!)
- Did not have any statistical analyses for anything in the study with the exception of the data shown in the figures (i.e., EDNOS patients took significantly less time to go into remission than AN patients)
Did BMI values change significantly after treatment for those who remitted from AN? Where the supposed decreases in anxiety, depression, and obsessionality following treatment scores significant? Did the quality of life improve in the patients who remitted? We don’t know. There’s ZERO information on any of this.
Sigh. In the next post I’m going to critically analyse the dozen or so reasons why Bergh et al. claim that eating disorders are not mental disorders (or, in their words, “have no underlying psychopathology”). Join me for the ride.
Bergh C, Callmar M, Danemar S, Hölcke M, Isberg S, Leon M, Lindgren J, Lundqvist A, Niinimaa M, Olofsson B, Palmberg K, Pettersson A, Zandian M, Asberg K, Brodin U, Maletz L, Court J, Iafeta I, Björnström M, Glantz C, Kjäll L, Rönnskog P, Sjöberg J, & Södersten P (2013). Effective treatment of eating disorders: Results at multiple sites. Behavioral neuroscience, 127 (6), 878-89 PMID: 24341712