Feedback Errors in Speech Therapy

I have been spending hours reviewing video of student SLPs (SSLPs) conducting speech therapy sessions, looking for snippets to take to my upcoming talks at ASHA2018. The students are impressively skilled with a very difficult CAS population but after this many hours of watching, repeated examples of certain categories of errors pile up in the provision of feedback to children about their attempts to produce the targets words, phrases and sentences. I am going to provide some examples here with commentary. In no way am I meaning any disrespect to the students because it is my experience that the average person becomes an idiot when a camera is pointed at them. I recall hearing about studies on the “audience effect” as an undergraduate – the idea is that when your skills are shaky you get worse when someone is watching but when your skills are excellent an audience actually enhances them. My social psychology prof said this even works for cockroaches! I can’t vouch for that but it certainly works for speech pathologists. I remember one time video-taping a session that was required for a course – I thought it went really well so I gave a copy to the parents and the course instructor. Later when watching it I could see clearly that for the whole half hour the child was trying desperately and without success to tell me that I was calling him by the wrong name (I had mixed him up with his twin brother whom I was also treating). I was oblivious to this during the live session but it was clear on the video. Anyway, these examples are not reflections on the students’ skill levels overall but they are examples of common feedback errors that I see in novice and experienced SLPs. Interestingly the clinical educators (CEs) who were supervising these sessions rarely mentioned this aspect of the students’ practice. Readers may find this blog useful as a template for reviewing student practice.

Category 1: No feedback

Child: [repeats 5 different sentences containing the target /s/ cluster words]

SSLP: [Turns to CE.] “What did you get?” [This is followed by 1 minute and 40 seconds of conversation about the child’s level of accuracy and strategies to improve it on the next block of trials.]

SSLP: [Turns back to child.] “You need to sit up. You got 2 out 5 correct. Now we’re going to count them on my fingers…”

Child: “Do we have to say these?”

Comment on vignette: In this case the SSLP did finally give feedback but too late for it to be meaningful to the child and after the telling the child off for slouching in her chair! Other variants on this are taking notes about the child’s performance or turning to converse with the child’s parent or getting caught up in the reinforcement game and forgetting to provide feedback. In CAS interventions it is common to provide feedback on a random schedule or to provide summative feedback after a block of trials. However, the child should be able to predict the block size and have information about whether their performance is generally improving or not. Even if the child does not have a count of number or percent trials correct, the child should know that practice stimuli are getting more difficult, reflecting performance gains. Sometimes, we deliberately plan to not provide feedback because we want the child to evaluate his or her own productions, but in these cases the child is told beforehand and the child is given a means of explicitly making that judgment (e.g., putting token in jar). Furthermore, the SSLP would be expected to praise the child for making accurate self-judgments or self-corrections. When the child does not get feedback or cannot track their own progress they will lose interest in the activity. It is common for SSLPs to change the game thinking that it is not motivating enough but there is nothing more motivating than a clear sense of success!

Possible solutions: Video record sessions and ask students to watch for and count the frequency of events in which the child has not received expected feedback. Provide child with visual guides to track progress indexed either as correct trials or difficulty of practice materials.

Category 2: Ambiguous feedback

SSLP: “Say [ska].”

Child: “[skak]”

SSLP: “OK, take the fish out.”

Comment on the vignette: In this case it is not clear if the SSLP is accepting the inexact repetition of her model. In our CAS interventions we expect the child to produce the model exactly because metathesis and other planning errors are common and therefore I would consider this production to be incorrect. Other ambiguous feedback that I observed frequently were “Good try” and “Nice try” and similar variants. In these cases the child has not received a clear signal that the “try” was incorrect. Another version of ambiguous feedback is to comment on the child’s behavior rather than the child’s speech accuracy (e.g., “You did it by yourself!” in which case the “it” is ambiguous to the child not clearly related to the accuracy of the child’s speech attempts).

Possible solutions: SSLPs really do not like telling children that have said something incorrectly. Ask students to role play firm and informative feedback. Have the students plan a small number of clear phrases that are acceptable to them as indicators of correct and incorrect responses (e.g., “I didn’t hear your snake sound” may be more acceptable than “No, that’s wrong”). Post written copies of the phrases somewhere in the therapy room so that the SLP can see them. Track the use of vague phrases such as “nice try” and impose a mutually agreed but fun penalty for exceeding a threshold number (buy the next coffee round for example). This works well if students are peer coaching.

Category 3: Mixed signals

SSLP: “Say [ska].”

Child: “[s:ka]”

SSLP: “Good job! Take the fish out.” [Frown on face].

Comment on the vignette: I am rather prone to this one myself due to strong concentration on next moves! But it is really unhelpful for children with speech and language delays who find the nonverbal message much easier to interpret than the verbal message.

Possible solutions: It would be better if SLP therapy rooms looked like a physiotherapy room. It annoys the heck out of me when we can’t get them outfitted with beautiful wall to ceiling mirrors. The child and SLP should sit or stand in front of the mirror when working on speech. Many games can be played using ticky tack or reusable stickers or dry erase pens. The SLP will be more aware of the congruence or incongruence between facial expressions, body language and verbal signals during the session.

Category 4: Feedback that reinforces the error

SSLP: “Repeat after me, Spatnuck” [this is the name of a rocket ship in nonsense word therapy].

Child: “fatnuck”

SSLP: “I think you said fatnuck with a [f:] instead of a [s:].

Comment on the vignette: Some SSLPs provide this kind of feedback so frequently that the child hears as many models of the incorrect form as the correct form. This is not helpful! This kind of feedback after the error is not easy for young children to process. To help the child succeed, it would be better to change the difficulty level of the task itself and provide more effective support before the next trial. After attempts, recasting incorrect tries and imitating correct tries can help the child monitor their own attempts at the target.

Possible solutions: Try similar strategies as suggested for ambiguous feedback. Plan appropriate feedback in advance. Plan to say this when the incorrect response is heard: “I didn’t hear the snake sound. Let’s try just the beginning of the word, watch me: sss-pat.” And when “spat” is achieved, plan to say “Good, I heard spat, you get a Spatnuck to put in space.”

Category 5: Confused feedback

SSLP: “Oh! Remember to curl your tongue when you say shadow.”

SSLP: “Oh! You found another pair.”

Child: “It’s shell [sʷɛo].”

SSLP: “Oh! I like the way you rounded your lips. Where is your tongue? Remember to hide your tongue.”

SSLP: Oh! You remembered where it was. You found another pair.”

Child: “Shoes [sʷuz].”

SSLP: “Oh! I like the way you rounded your tongue.”

Comment on vignette: In this vignette the SSLP is providing feedback about three aspects of the child’s performance-finding pairs when playing memory, rounding lips when attempting “sh” sounds, and in some cases anterior tongue placement when attempting the “sh” sound as well. One aspect of her feedback that is confusing when watching the video is the using of the exclamation “Oh!” Initially it appeared to signal an upcoming correction but it became so constant that it was not a predictable signal of any kind of feedback and was confusing. The exclamation had a negative valence to it but it might precede a correction or positive feedback. The SSLP confused her feedback about lips and tongue and it was not clear whether she was expecting the child to achieve the correct lip gesture, the correct tongue gesture or both at the same time.

Possible solutions: This can happen when there is too much happening in a session. The CE could help the SSLP restructure the session so that she can focus her attention on one aspect of the child’s behavior at a time, like this: “I want you to name these five pictures. Each time I am going to watch your lips. When you are done you can put the pictures on the table and mix them up for our game later.” If the child rounds the lips each time, switch to focusing on the tongue. When the ten cards are on the table play memory, modeling the picture names. In this way the three behaviors (rounding lips, retracting tongue, finding pairs) are separated in time and the SSLP can focus attention on each one with care, providing appropriate feedback repeatedly during the appropriate intervals.

Category 6: Confused use of reinforcement materials

SSLP: “Repeat after me, [ska].”

Child: “[θak]”

SSLP: [ska]

Child: “[θak]”

SSLP: “OK, take the fish out.”

SSLP: “Repeat after me, [ska].”

Child: [ska]

SSLP: “There you got it, take the fish out.”

SSLP: “Repeat after me, [ska].”

Child: [ska]

SSLP: “Good, and the last one, [ska].”

Child: [ska]

SSLP: “That’s good, take the fish out.”

Comment on vignette: In this vignette the child cannot tell if he gets a fish for correct answers or wrong answers or any answer. It is even worse if the child has been told that he will get a fish for each correct answer. Sometimes a student will say “Everything was going fine, we were having fun and then he just lost it!” When you look at the video you see exchanges such as the one reproduced here leading up to a tantrum by the child. The SSLP has broken a promise to the child. They don’t forgive that.

Possible solutions: This one is hard because it is a classic rooky mistake. Experience is the best cure. Reducing the number of tasks that the SSLP must do simultaneously may help. Therefore, in the early sessions the CE might keep track of the child’s correct and incorrect responses for the SSLP and allow her to focus on managing the materials and the child’s behavior. SSLPs would never think of this but it is possible to let the child manage the reinforcement materials themselves in some cases. One of our favorite vignettes, reprinted on page 463 of DPD2e (Case Study 9-4) involved an error detection activity in which the child could put toy animals in the barn but only when the SSLP said the names of the animals correctly. The child had the toys in his hands throughout the activity. He would not put them in the barn unless the clinician said the words correctly and would get annoyed if she said them wrong, telling her “you have to say cow [kau]!” SSLPs can learn that it is not necessary to control everything.

I put these here for students and clinical educators and speech-language pathologists and hope that you will have fun finding these feedback mishaps in your own sessions. If you come up with better strategies to avoid them than I have suggested here please share them in the comments.

Advertisements

Which SLPs are Effective?

In my last two blog posts I have been talking about how to ensure that your speech therapy program “works”, in other words, how can you be sure that what you do is effecting change in your patient over and above the change that would occur due to maturation and history effects alone? I have suggested that if you choose treatment approaches that have been validated via randomized controlled trials as effective approaches and if you demonstrate that your patient is improving you can be reasonably sure that you are having a positive effect on your patient. I have further cautioned that you need to read the original research carefully and implement the treatment approaches in accordance with the treatment efficacy trials with respect to procedures, treatment intensity and so on in order to ensure that you will get the same effect. These details – the treatment procedures that you decide to implement with your patient – are referred to as the specific ingredients of your treatment program. Throughout my research career I have been focused on the relative efficacy of these specific ingredients – is it effective to use perception training or stimulability training or prepractice with visual cues in comparison to usual care? For example, one-year follow-up of the children treated in Rvachew, Nowak and Cloutier (2004) showed that 50% of the children who received usual care + speech perception training started school with normalized speech versus only 19% of the children who received usual care + dialogic reading. I obviously feel that an important role of the SLP is to know the scientific literature and choose the right specific ingredients for their patients.

In contrast, Ebert and Kohnert (2010) point out that the effectiveness of speech therapy might also be due to “common factors” which include (following Grencavage and Norcross): the patient, the clinician, and the patient-clinician alliance, change processes, and treatment structure. Studies on the effectiveness of teachers and psychotherapists are starting to appear with increasing frequency but I am not aware of any published systematic studies of SLP effectiveness that take a “common factors” approach. Ebert and Kohnert re-analyzed the data from one of my studies (Rvachew and Nowak, 2001, discussed in my previous blog) and concluded that although target selection strategy accounted for a larger proportion of variance in outcomes, individual differences in clinician effectiveness accounted for 20% of variance in outcomes. These researchers surveyed SLPs in Minnesota and asked them to rate various factors for their importance in determining client outcomes. The results showed that SLPs weight client-clinician factors very highly with “rapport” being the item rated as having the greatest impact on therapeutic outcomes. Recently Geraldine Wotton wrote a blog post on the power of the therapeutic relationship that expresses this commonly held view. The thing is however, I knew the SLPs who provided the intervention in Rvachew and Nowak and I can tell you that there were no discernible differences in rapport between these SLPs and their clients. Furthermore, at the time I was the research coordinator for allied health in the hospital and I was responsible for the client satisfaction questionnaire in the hospital. Families reported high levels of satisfaction with their clinicians while reporting varying levels of satisfaction with their child’s outcomes. I was always impressed by the fact that parental satisfaction with their child’s speech outcomes and objective measures of child outcomes were highly correlated (given that I was running several RCTs at the time I could look at this) but uncoupled from uniformly high satisfaction ratings for their relationship with the therapist. I certainly agree that the strong positive relationship between SLPs and their patients is an important factor in treatment efficacy – I just don’t agree that it explains variations in treatment outcomes: think about this carefully – SLPs are selected to have strong interpersonal skills and we are very good at establishing rapport with our patients but we do not all get the same results. There is something else going on here.

Françoise and I recently completed a RCT involving 72 francophone children in which the clinicians were student SLPs from McGill. We have 6 videorecorded therapy sessions for each child, representing more than a dozen student SLPs. Unfortunately we have run out of funds so we haven’t been able to analyze all the video but two students, Amanda Langdon and Hannah Jacobs, obtained summer research bursary funds from the Faculty of Medicine to conduct a pilot project in which they coded the videos for 6 student clinicians, attempting to identify common factors that might differentiate between more and less effective SLPs. In this case the supervising clinical educators told us which student SLPs were more or less effective in their opinion, rating them as “accomplished” or “struggling”. Then Hannah and Amanda coded the videos for factors related to the clinician, the clinician-child alliance and to change processes. Interestingly the factors that differentiated “accomplished” versus “struggling” student SLPs were not those that would be ascribed to the “clinician” category in Glencavage and Norcross’ model. Rather we found large differences in variables that could be categorized as “change processes”. In Glencavage and Norcross’ paper a lot of the factors categorized as change processes are specific to psychotherapy but some are common to speech therapy as well, for example “acquisition and practice of new behaviors”, “provision of a therapeutic rationale”, “naming the problem”, and “contingency management”. Applied to speech therapy we can hypothesize that SLPs may vary in their ability to communicate and/or negotiate the goals of the therapy program to or with the patient, maintain a high response rate during sessions so as to ensure that most of the session is spent practicing new behaviors, and manage contingencies so that the patient is receiving appropriate feedback about their responses during practice. We observed changes in these skills across the six week treatment program for student SLPs who were rated to be “accomplished” or “struggling” by their supervising clinical educators. We found that all the students increased the amount of time devoted to direct therapy in their sessions during the course of their practicum. Accomplished students began with good contingency management skills and improved those skills to an even higher level after six weeks of practice. On the other hand, struggling students began and ended the practicum with poor contingency management skills – in particular these students did not provide appropriate feedback after incorrect responses by their clients. Interestingly, in comparison to struggling students, accomplished students spent more rather than less time in “off task” behavior which may mean that they had more resources available for conversation that served to establish rapport with their clients. Struggling students spent a lot of time “manipulating materials” and therefore their disorganized approach to the therapy sessions may have interfered with the SLP-client alliance. Unfortunately this study is tiny; the coding is hugely time consuming and expensive. However I think that it is crucial for our profession that resources be expended to study these therapeutic processes and the means to improve our students’ skills in learning these skills during their preprofessional practice.

I’d love to hear from student SLPs about your experiences with learning these skills. What do you think your clinical educators could do to help you learn these skills? I’d also love to hear from practicing SLPs – do you agree that skill in the engagement of change processes is an important factor in therapeutic effectiveness? Which change processes do you think are most important in speech therapy?