Let's talk about research...

On the VitalStim website, a "results poster" for a study entitled "Experience with transcutaneous electrical stimulation: a new treatment option for the management of pediatric dysphagia” is posted. I can only assume that this one of the studies the company refers to when it claims (see VitalStim FAQs) that “Clinical studies showed VitalStim Therapy as effective for patients 1-4 years old as for older patients. VitalStim Therapy was also found to be effective on infants less than 1 year old.”

This particular study was presented at the North Carolina Speech and Hearing Meeting in 2004. It has not been published. Granted, this "results poster" only summarizes the study results, but it raises a number of questions.

• There is no control group mentioned. How do you expect to "report on the effectiveness of this treatment modality" without a control group? This is Research 101.
• There is no mention of blinding. This raises the possibility of examiner bias.
• Questionable patient selection. Study subjects were "a consecutive sample of 30 pediatric patients who were referred for ES therapy after failing conventional therapy." Age range: 2 weeks to 16 years. Now tell me, how does a 2-week-old infant even have time to "fail conventional therapy"?
• A variety of etiologies are lumped together. This wouldn't necessarily be bad, except the characteristics of the dysphagia are not specified, so you can't really tell if the researchers are comparing apples with apples.
• Non-validated evaluation instrument. Modified Barium Swallow evaluation was used to generate "swallow scores." Last I checked, this "scoring system" has not been researched or validated. As far as I can tell, it was invented for the original VitalStim study . The scale goes like this:

  O No swallow
  1 Swallow saliva only
  2 Tastes of any consistency safe
  3 Puree texture safe
  4 Honey consistency safe
  5 Nectar consistency safe
  6 Milk consistency safe
  7 All consistencies including water safe

  This generates its own set of questions. What is "safe"? No aspiration? Trace aspiration? Less than 10% of the bolus aspirated? No pharyngeal residue? And that whole "swallow saliva only" category -- rather iffy to test that on MBS, which usually elicits volitional swallows, which are very different than the way we usually manage saliva. And what about a patient that does swallow his/her saliva, but aspirates most of it? What score do they get, since they don't have an absent swallow, but they are presumably not "safe" to swallow saliva? 0.5?
  

In addition to the methodological concerns, the results and conclusions prompt a number of other questions.

• "Swallow scores improved in 17 (57%) of the patients..." How improved is "improved"? I assume one point difference in swallow score would constitute an "improvement." That is not a large change, merely one step up in the texture/consistency hierarchy created by the "swallow scores" classification. I would venture to guess that some of those "improvements" could really be due to varied performance from one MBS session to the next, rather than true physiological improvement.
• "The average final swallow score of those who improved demonstrated safety on puree and honey consistency liquids." This is not all that impressive. What was the average starting swallow score? (I believe it was 2, if I am interpreting the cryptic "Change in Swallow Score" table correctly). At any rate, for a score to be "average," that means some scores were better and some scores were worse -- some patients, despite improving, were still not able to tolerate even purees. And the patients who achieved a normal swallow? What were their starting scores? Could it be that they were less impaired to begin with? Those "averages" don't really tell us much.
• Let's look at the numbers for those who improved a little more closely. 5/30 patients (17%) achieved a normal score (Swallow Score = 7), and a total of 17/30 patients (57%) improved. This means that there were 12 patients who improved, but did not attain a normal score. We will infer that the average score for all patients who improved was 3.5 (halfway between "puree texture safe" and "honey consistency safe", as the results state that the average score "demonstrated safety on puree and honey consistency liquids." Now, bear with me -- statistics are not my forte -- but if 5/17 patients who improved had a normal score of 7, I believe that means that the average score of the 12 remaining must come to 2.05 (that is, [(5x7) + (12x2.05)]/17 = 3.5), which is only at a "tastes of any consistency safe" level. Not impressive at all. Even if we are generous and assume that the average score for all patients who improved was 4, the average score of the 12 remaining patients is still only 2.75. Those 5 perfect scores sure help skew the averages!
• "The majority of those with dysphagia of unknown etiology and acquired CNS lesions improved." Hmmm. From the chart it looks like "Unknown" was actually 50/50 in terms of improvement, which is NOT a majority. (Now, if you combine it with "Acquired," it is, but that's just massaging the numbers til they say what you want.)

The most unfortunate part about this whole thing is that many SLP's do not take the time to analyze studies in this way. They take results of studies at face value, without analyzing their quality. It is not enough to clamor for "evidence" and then take whatever comes down the pike without questioning.