Which SLPs are Effective?

In my last two blog posts I have been talking about how to ensure that your speech therapy program “works”, in other words, how can you be sure that what you do is effecting change in your patient over and above the change that would occur due to maturation and history effects alone? I have suggested that if you choose treatment approaches that have been validated via randomized controlled trials as effective approaches and if you demonstrate that your patient is improving you can be reasonably sure that you are having a positive effect on your patient. I have further cautioned that you need to read the original research carefully and implement the treatment approaches in accordance with the treatment efficacy trials with respect to procedures, treatment intensity and so on in order to ensure that you will get the same effect. These details – the treatment procedures that you decide to implement with your patient – are referred to as the specific ingredients of your treatment program. Throughout my research career I have been focused on the relative efficacy of these specific ingredients – is it effective to use perception training or stimulability training or prepractice with visual cues in comparison to usual care? For example, one-year follow-up of the children treated in Rvachew, Nowak and Cloutier (2004) showed that 50% of the children who received usual care + speech perception training started school with normalized speech versus only 19% of the children who received usual care + dialogic reading. I obviously feel that an important role of the SLP is to know the scientific literature and choose the right specific ingredients for their patients.

In contrast, Ebert and Kohnert (2010) point out that the effectiveness of speech therapy might also be due to “common factors” which include (following Grencavage and Norcross): the patient, the clinician, and the patient-clinician alliance, change processes, and treatment structure. Studies on the effectiveness of teachers and psychotherapists are starting to appear with increasing frequency but I am not aware of any published systematic studies of SLP effectiveness that take a “common factors” approach. Ebert and Kohnert re-analyzed the data from one of my studies (Rvachew and Nowak, 2001, discussed in my previous blog) and concluded that although target selection strategy accounted for a larger proportion of variance in outcomes, individual differences in clinician effectiveness accounted for 20% of variance in outcomes. These researchers surveyed SLPs in Minnesota and asked them to rate various factors for their importance in determining client outcomes. The results showed that SLPs weight client-clinician factors very highly with “rapport” being the item rated as having the greatest impact on therapeutic outcomes. Recently Geraldine Wotton wrote a blog post on the power of the therapeutic relationship that expresses this commonly held view. The thing is however, I knew the SLPs who provided the intervention in Rvachew and Nowak and I can tell you that there were no discernible differences in rapport between these SLPs and their clients. Furthermore, at the time I was the research coordinator for allied health in the hospital and I was responsible for the client satisfaction questionnaire in the hospital. Families reported high levels of satisfaction with their clinicians while reporting varying levels of satisfaction with their child’s outcomes. I was always impressed by the fact that parental satisfaction with their child’s speech outcomes and objective measures of child outcomes were highly correlated (given that I was running several RCTs at the time I could look at this) but uncoupled from uniformly high satisfaction ratings for their relationship with the therapist. I certainly agree that the strong positive relationship between SLPs and their patients is an important factor in treatment efficacy – I just don’t agree that it explains variations in treatment outcomes: think about this carefully – SLPs are selected to have strong interpersonal skills and we are very good at establishing rapport with our patients but we do not all get the same results. There is something else going on here.

Françoise and I recently completed a RCT involving 72 francophone children in which the clinicians were student SLPs from McGill. We have 6 videorecorded therapy sessions for each child, representing more than a dozen student SLPs. Unfortunately we have run out of funds so we haven’t been able to analyze all the video but two students, Amanda Langdon and Hannah Jacobs, obtained summer research bursary funds from the Faculty of Medicine to conduct a pilot project in which they coded the videos for 6 student clinicians, attempting to identify common factors that might differentiate between more and less effective SLPs. In this case the supervising clinical educators told us which student SLPs were more or less effective in their opinion, rating them as “accomplished” or “struggling”. Then Hannah and Amanda coded the videos for factors related to the clinician, the clinician-child alliance and to change processes. Interestingly the factors that differentiated “accomplished” versus “struggling” student SLPs were not those that would be ascribed to the “clinician” category in Glencavage and Norcross’ model. Rather we found large differences in variables that could be categorized as “change processes”. In Glencavage and Norcross’ paper a lot of the factors categorized as change processes are specific to psychotherapy but some are common to speech therapy as well, for example “acquisition and practice of new behaviors”, “provision of a therapeutic rationale”, “naming the problem”, and “contingency management”. Applied to speech therapy we can hypothesize that SLPs may vary in their ability to communicate and/or negotiate the goals of the therapy program to or with the patient, maintain a high response rate during sessions so as to ensure that most of the session is spent practicing new behaviors, and manage contingencies so that the patient is receiving appropriate feedback about their responses during practice. We observed changes in these skills across the six week treatment program for student SLPs who were rated to be “accomplished” or “struggling” by their supervising clinical educators. We found that all the students increased the amount of time devoted to direct therapy in their sessions during the course of their practicum. Accomplished students began with good contingency management skills and improved those skills to an even higher level after six weeks of practice. On the other hand, struggling students began and ended the practicum with poor contingency management skills – in particular these students did not provide appropriate feedback after incorrect responses by their clients. Interestingly, in comparison to struggling students, accomplished students spent more rather than less time in “off task” behavior which may mean that they had more resources available for conversation that served to establish rapport with their clients. Struggling students spent a lot of time “manipulating materials” and therefore their disorganized approach to the therapy sessions may have interfered with the SLP-client alliance. Unfortunately this study is tiny; the coding is hugely time consuming and expensive. However I think that it is crucial for our profession that resources be expended to study these therapeutic processes and the means to improve our students’ skills in learning these skills during their preprofessional practice.

I’d love to hear from student SLPs about your experiences with learning these skills. What do you think your clinical educators could do to help you learn these skills? I’d also love to hear from practicing SLPs – do you agree that skill in the engagement of change processes is an important factor in therapeutic effectiveness? Which change processes do you think are most important in speech therapy?


Don’t get tricked: Why it pays to read original sources.

In my last blog post I suggested that you can have confidence in the effectiveness of your clinical practice if you select treatment practices that have been validated by research. Furthermore, I provided links to some resources for summaries of research evidence. In this blog post I want to caution that it is important to read the original sources and to view the summaries, including meta-analyses, with some skepticism. Excellent clinical practice requires a deep knowledge of the basic science that is the foundation for the clinical procedures that you are using. Familiarity with the details of the clinical studies that address the efficacy of those procedures is also essential. I will provide two examples where a lack of familiarity with those details has led to some perverse outcomes.

Two decades ago it was quite common for children who were receiving services from publically funded providers in Canada to receive 16-week blocks of intervention. Then we went through the recession of the nineties and there was much pressure on managers in health care to cut costs. Fey, Cleave, Long, and Hughes (1993) conveniently published an RCT demonstrating that a parent intervention was just as effective as direct intervention provided by the SLP to improve children’s expressive grammar – the icing on the cake was that the parent-provided service required half as many SLP hours as the direct SLP-provided service. All across Canada, direct service blocks were cut to 8 weeks and parent-consultation services were substituted for the direct therapy model. About a decade after that I made a little money myself giving workshops to SLPs on evidence based practiced. The audiences were always shocked when I presented the actual resource inputs for Fey et al.’s interventions: (1) direct SLP intervention –  cost = 40 hours per child over 20 weeks, versus (2) parent administered intervention – cost = 21 hours per child over 20 weeks. So you see, the SLPs had been had by their managers! The SLPs would have been better positioned to resist this harmful change in service delivery model if they had been aware of the source of the claim that you could halve your therapy time by implementing a home program and get the same result. I don’t know that our profession could have changed the situation by being more knowledgeable about the research on service delivery models because the political and financial pressures at the time were extreme – but at least we and our patients would have had a fighting chance!

Another reason that you have to be vigilant is that the authors of research summaries have been known to engage in some sleight of hand. An example of this is chapter on Complexity Approaches by Baker and Williams in the book Interventions for Speech Sound Disorders in Children. This book is pretty cool because each chapter describes a different approach  and is usually accompanied by a video demonstration. Each author was asked to identify all the studies that support the approach and put them on a “levels of evidence” table. As indicated in a previous blog post, the complexity approach to selecting targets for intervention is supposedly supported by a great many studies employing the multiple probe design which is a fairly low level of evidence because it does not control for maturation or history effects. In the Baker and Williams “levels of evidence” table all of these single subject studies are listed  so it looks pretty impressive. The evidence to support the approach looks even more impressive when you notice that two randomized controlled trials are shown at a higher level on the table. This table leads you to believe that the complexity approach is supported by a large amount of data and the highest level of evidence until you realize that neither of those two RCTs, Dodd et al. (2008) and Rvachew and Nowak (2001), support the complexity approach. Even when you read the text, it is not clear that these RCTs do not provide support for the approach because the authors are a bit wafflely about this fact.  Before I noticed this table I couldn’t understand why clinicians would tell me proudly that they were using the complexity approach because it is evidence based. It is pretty hard to keep up with the evidence when you have to watch out for tricks like this!

In the comments to my last blog post there were questions about how you can be sure that your treatment is leading to change that is better than maturation alone. An RCT is designed to answer just that question so I am going to discuss the results of Rvachew and Nowak (2001), as detailed in a later paper, Rvachew, S. (2005). Stimulability and treatment success. Topics in Language Disorders. Clinical Perspectives on Speech Sound Disorders, 25(3), 207-219. Unfortunately this paper is hard to get so a lot of SLPs are not aware of the implications of our findings for the central argument that motivates the use of the complexity approach to target selection.  Gierut (2007) grounds the complexity approach on learnability theory, paradoxically the notion that language is essentially unlearnable and thus the structure of language must be innately built in. Complex language inputs are necessary to trigger access to this knowledge. Because of the hierarchical structure of this built-in knowledge, exposure to complex structure will “unlock the whole”, having a cascading effect down through the system. On the other hand, she claims that “it has been shown that simpler input actually makes language learning more difficult because the child is provided with only partial information about linguistic structure (p. 8).”

We tested this hypothesis in our RCT. Each child received a 15 item probe of their ability produce all the consonants of English in initial, medial and final position of words. The phonemes that they had not mastered were then ordered according to productive phonological knowledge and developmental order. Michele Nowak selected potential treatment targets for each child from both ends of the continuum. I independently (blindly, without access to the child’s test information or knowledge of the targets that Michelle had selected) randomly assigned the child to treatment condition, either ME or LL. ME condition means that the child was treated for phonemes for which the child had most knowledge and which are usually early developing. LL condition means that the child was treated for phonemes for which the child had least productive phonological knowledge and which are usually late developing. The children were treated in two six week blocks with a change in treatment targets for the second block using the same procedure to select the targets. The figure below shows probe performance for several actual and potential targets per child: the phoneme being treated in a given block, the phoneme to be treated in the next block (or that was treated in the previous block) and the phonemes that would have been treated if the child had been assigned to the other treatment condition. As a clinician, I am interested in learning and retention of the treated phonemes, relative to maturation. As a scientist who is testing the complexity approach, Gierut is interested in cross-class generalization, regardless of whether the child learns the targeted phoneme. We can look at these two outcomes across the two groups.

Let’s begin with the question of whether the children learned the target phonemes and whether there is any evidence that this learning is greater than what we would see with maturation alone. In the chart, learning during treatment is shown by the solid lines whereas dotted lines indicate periods where those sounds were not being treated. A1 is the assessment before the first treatment block, A2 is the assessment after the first block and before the second block, and A3 is the last assessment after the second treatment block. On the left hand side, we see that the ME group was treated during the first block for phonemes that were mastered in one word position but not in the other two (average score of 6/15 prior to treatment). The slopes of the solid versus dotted lines show you that change from A1 to A2 was greater than change from A2 to A3. This means that these targets showed more change when they were being treated in the first block than when they were not being treated during the second block. During the second block, we treated slightly harder sounds that were not mastered in any word position, with a starting probe score of 3/15 on average. These phonemes improved from A1 to A2 even though they weren’t being treated but the rate of improvement is much higher between A2 and A3 when they were being treated. Interestingly, the slopes of the solid lines and the slopes of the dotted lines are parallel – this is your treatment effect – this is the proof that treatment is more effective than not treating. As further proof we can look at the results for the LL group. We have a similar situation with parallel solid and dotted lines for the phonemes that were treated in the first and second blocks at the bottom of the chart. We don’t have as much improvement for these phonemes because they were very difficult, unstimulable late developing sounds (targets that are consistent with the complexity approach). None-the-less the outcomes are better while the phonemes are being treated than when they are not (in fact there are slight regressions during the blocks when these sounds are not treated). At the same time, the phonemes for which the children have the most knowledge improve spontaneously (Gierut would attribute this change to cross-class generalization whereas I attribute this change to maturation). The interesting comparison however is across groups. Notice that the ME group shows a change of 4 points for treated “most knowledge” phonemes versus a change of 3 points for the untreated “most knowledge” phonemes for the LL group. This is not a very big difference but none-the-less, treating these phonemes results in slightly faster progress than not treating them.

In our 2001 paper we reported that progress for treated targets was substantially better for children in the ME condition than for children in the LL condition (in the latter group, the children remained unstimulable for 45% of targets after 6 weeks of therapy). However, the proponents of the complexity approach are not interested in this finding. If the child does not learn the hard target that is an acceptable price to pay if cross-class generalization occurs and the child learns easier untreated phonemes. If you look at the right hand side of the chart by itself, the chart can be taken as support for the complexity approach because spontaneous gains are observed for the “most knowledge” phonemes. The problem is that the proponents of this approach have argued that exposure to “simpler input actually makes language learning more difficult” – it is literally supposed to be impossible to facilitate learning of harder targets by teaching simpler targets. Therefore the real test of the complexity approach is not in the right hand chart. We have to compare the rate of change for the unstimulable targets across the two groups. It is apparent that the gain for UNTREATED unstimulable phonemes (ME group, gain = 2) is double that observed for TREATED unstimulable phonemes (LL group, gain = 1). The results shown on the left clearly show that treating the easier sounds first facilitated improvements for the difficult phonemes. I have explained this outcome by reference to dynamic systems theory in Rvachew and Bernhardt (2010). From my perspective, it is not just that my RCT shows that the complexity approach doesn’t work. It’s that my RCT is just part of a growing and broad based literature that invalidates the “learnability approach” altogether. Francoise and I describe and evaluate this evidence while promoting a developmental approach to phonology in our book Developmental Phonological Disorders: Foundations of Clinical Practice.


Probe Scores for Treated and Untreated Phonemes

Probe Scores for Treated and Untreated Phonemes









The larger point that I am trying to make here is that SLPs need to know the literature deeply. The evidence summaries tend to take a bit of a “horse race” approach, grading study quality on the basis of sometimes questionable checklists and then making conclusions on the basis of how many studies can be amassed at a given level of the evidence table. This is not always a clinically useful practice. It is necessary to understand the underlying theory, to know the details of the methods used in those studies, and to draw your own conclusions about the applicability of the treatments to your own patients. This means reading the original sources. In order to achieve this level of knowledge we need to reorganize our profession to encourage a greater number of specialists in the field because no individual SLP can have this depth of knowledge about every type of patient that you might treat. But it should be possible to encourage the development of specialists who are given the opportunity to stay current with the literature and provide consultation services to generalists on the front lines. Even if we could ensure that SLPs had access to the best evidence as a guide to practice however, there are some “common factors” that have a large impact on outcomes even when treatment approach is controlled. In my next post I will address the role of the individual clinician in ensuring excellent client outcomes.