Decoding Success Rate Myths

David Condos: Welcome to this episode of Recovery Unscripted. I’m your host David Condos, and this podcast is powered by Foundations Recovery Network. On this episode, we’ll be talking with Siobhan Morse who’s the Division Director of Clinical Services for FRN as part of Universal Health Services. She’s also spearheaded a number of new research studies related to the treatment of co-occurring substance use and mental health disorders including one about myths that surround the term success rates, which we’ll be talking about on the show today. Let’s get started.

[music]

David: Welcome, Siobhan. Good to have you here.

Siobhan Morse: Thank you so much for having me.

David: Absolutely. How are you doing today?

Siobhan: I’m doing great. Thanks for asking.

David: Excellent. I thought we’d get started by having you talk a little bit about how you began in the clinical research industry and with addiction mental health treatment.

Siobhan: I spent, let’s say, about eight years working in clinical pharmacology research. That’s research basically with a placebo or this drug versus that drug and I got a background in understanding how research works. From there, as I was at the university, I began working with the editor of The Journal of Consulting and Clinical Psychology. We ran an NIMH-sponsored demonstration project for the government, for mental health. That was a really big project looking at public research, looking at outcomes based on the provision of interventions in a large setting.

My research career followed this path of expanding and then I worked a little bit with the US Department of Education in the corporate academy schools with our risk youths and did some community projects. Including I worked with United Way and Children Now on developing a report card on the status of children services for Dade County, Florida, where I was living at that time.

That’s the research piece of how my career progressed. Let’s see, in 2006, I began working in addiction treatment and, let’s see, over the next three years in addiction treatment I got a master addiction counseling certification, and I became an Arise Interventionist as well. When I got to Foundations I was able to blend all those disparate pieces of my background.

David: Nice, yes. Now you’re at Foundations, could you tell us a little bit more about your current role?

Siobhan: Absolutely. I am the Division Director of Clinical Services. What that includes is the research piece which is we measure outcomes in our patients. We look at the comparison of where they are on certain indicators such as psychological health and their addiction, their substance use disorder, medical health, the quality of their life, their ability to manage their medications. Just a whole world of indicators, both at intake, at 30 days after discharge, six months after discharge, and one year after discharge. We can see how well we’re doing, we can feed that information back into our clinical programming and we can help support the industry as well.

David: What, off the top of your head, is something that you enjoy most about where you are right now, what you’re doing?

Siobhan: One of the things I get to do is, on behalf of Foundations, present new information and teach people in the industry how to set up programs such as our research program and that is probably my favorite. That and as a part of my clinical position, I’m the holder of this system of care for Foundations. Our system of care is who we are and we continue to aspire to be with each individual patient. That part of my job is such an honor and I love being able to do that.

David: Cool, yes. One of the most current research pieces that you’ve done is about success rates and the myths surrounding them. I’d like to talk about that for a second. One way that this type of research is used by treatment centers obviously is to show the success rates but how would you say it can be dangerous for these treatment programs to rely too heavily on those.

Siobhan: There’s a couple of factors in this. First, let’s just even look at the words “success rates.” We don’t define success. It’s so important to understand that the patient, their family, they’re the ones who define, “What is the success for me?” That’s just the foundation and the basis of what patient-centered care looks like, including the patient in the planning and what they aspire to. To use the word success rates has no real clear definition so that would probably be the first thing I would say to people looking at websites. When you see a big “our success rates are…,” that would be my first red flag because I’m going to tell you what it is I’m telling you.

If I’m telling you what our abstinent rate is, it’s going to say “abstinence rates,” right? If I’m telling you our substance use outcomes by days of use, it’s going to say “substance use.” “Success rates” is fairly ambiguous so that would be the first piece of that. Then secondly, really getting a clear understanding on how these numbers, once you figure out what the numbers mean, how were they generated? How were these developed? Were they developed using the scientific method? In what percentage of the population were they developed? How many people do they represent? Is this fair to say it’s typical, a typical representation?

I mean, SAMHSA, the Substance Abuse and Mental Health Services Administration in Washington, has some standards with regard to how many people should be enrolled in an evaluation of a treatment center or of a program, as well as how many people need to be reached after completion of the program to have fair and reasonably relied and valuable results. You want to make sure that whatever numbers you’re looking at meet those standards and that they truly are representative. To have a success result that, say, an 85% success rate that really only represents 40% of your population is not giving you typical results. That’s a big part of what we were talking about in that recent white paper.

David: Nice, yes. That’s a good point. The first part of that you said is it does seem really arbitrary for a facility to determine that this is what success looks like and that should apply to everyone across the board when it really doesn’t work that way.

Siobhan: No, it doesn’t. You see this often as well, you see many places talking about individualized treatment. Well, individualizing treatment means also individualizing the goals of treatment. Not everybody has the same goals. The 18-year-old male coming into treatment may have very different goals from the 45-year-old female. So it’s taking the time to sit with them, with their families, with the community that may have referred them and coming up with appropriate and clear and meaningful goals for their treatment so that you can get success in that way for them.

[music]

David: With your research team, what does the process look like when you’re collecting this outcomes data?

Siobhan: Every person who presents for treatment at a Foundations Recovery Network site is offered the opportunity to participate in this research project. The way that they’re offered that opportunity is they’re given a separate “consent for research” form that’s been approved by our institutional review board that’s registered with federally with the Department of Health and Human Services in Washington. They sign the consent and they give us information on both the clinical intake information as well as information so that we can reach them, locator forms, after treatment.

Then, that information will be shared with my research team and so at that point, everything that is collected by my research team after that intake is now research health information which is held to an actual higher standard than personal health information, in that I do not share it back on an individual basis with the treatment center. That information is completely confidential within the research team.

When we call our patients because we’re able to do that, we’re not trying to readmit them, we’re just trying to collect the information and so they know they can be honest with us and they can tell us the truth about what’s really going on in their lives. At 30 days, six months, and one year we will reach out to these patients and we’ll collect the same information that we collected at intake so that we can see their progress over time.

David: Yes, that was the next thing I was going to ask about. You collect at one month, six months, 12 months. Why would you say that breadth of data is important?

Siobhan: There’s a series of questions within the data that actually says in the last 30 days have you or did you so the one month just kind of gives us since the end of treatment or since discharge and then six months is a pretty important time point. Now you’re deep back into your life, right? What we wanted to be able to do is to measure that half year point, see how things are going, and then bringing it out to the year just simply because people who have achieved a full year of sobriety or of remission are much more likely to continue to experience similar outcomes in their life.

David: What would you say are some challenges facing the research community in this industry?

Siobhan: In all honesty, the first challenge is the lack of a research community in this industry. Yes, there’s a world of academia and, say, research that lives really separate from what’s going on in everyday life in treatment centers. Really, the way that the treatment system looks at this point is you have a public sector treatment system and that’s funded by state and government entities primarily.

Those people are responsible for collecting that SAMHSA information that we mentioned. If they receive funding, then they have to collect some form of evaluation or outcome. The rest of the world, all of us private entities, there is no requirement. There’s not going to be direct revenue associated with it so it’s hard, especially for smaller entities, to build this into their plan.

David: Sure, yes. It’s tough to justify that when you have so much other things.

Siobhan: Exactly. What I’ve been sharing in my talks and what I’ve been sharing is first there are case management programs that can supplement and help, that are available so that you’re both providing a service of case management over time but also through that service you could collect results and begin to look and see how things are going. Some small facilities utilize that method. Even if the only opportunity you have to reach out to patients after they leave your facility is the alumni group, then use it.

For me, I believe it’s about starting somewhere. It’s about building in some level of accountability and actually learning how to use the data. This is a very data-rich environment even without the follow-up phone calls. A lot to be learned and to begin to just sort of get everyone in the industry facing that direction and saying, “Well, the data is useful for me in treatment planning for this patient but what if I pulled it all together? What trends, what information can I pull that’s going to help me serve the larger audience in the industry in general?”

David: Yes, and just to start somewhere.

Siobhan: Exactly.

David: It’s better than nothing.

Siobhan: Agreed.

David: What are some third-party bodies that oversee both the research side and the treatment side?

Siobhan: There’s a very little regulation in our industry. What we do is we have the IRB, their primary purpose is to protect the patient’s safety and protect the patient rights. Secondarily, I utilize my IRB as also to say, “Okay, here’s what I did. Here’s the data I collected. Here’s the conclusions I drew. Does this make sense? Is this logical? Are my conclusions not out in left field hopefully?” and they give me guidance in that.

Additionally, one of the best ways to get third party validation of your research or your findings is to publish. I’m using scientific peer reviewed journals. You have to go through a process where a blind reviewer looks at what you’re saying and says, “Yes, this is valid, this makes sense,” and in effect, literally picks it apart. And so in that span, I believe we have 13 publications listed on the website at this point with our original data. That’s been a big part of what we’ve been able to accomplish here so that just to further, it’s a great contribution to the field because few people are doing it in our industry with our specific population. Secondarily, it also does further validate the results that we’re able to say we have.

David: Sure, and for people who don’t know, what is an IRB and what is its role?

Siobhan: An IRB is an Institutional Review Board, right? They are again registered with the Department of Health and Human Services up in D.C. federally and it’s a group of both experts and laypeople. For example, I have an attorney, I have someone in recovery, I have several professors of social work, psychology, nursing. We get together anywhere from two to four times a year depending on the need to review documentation, to review what my research process is and then to review the results.

[music]

David: I’d like to transition a little bit now to how all of this affects the patients. The treatment centers in this field use these success rates to make a lot of promises in their marketing. How would you advise potential patients and their families to sort through all of that and decipher the truth out of these promises?

Siobhan: I guess the first thing that brings to mind is transparency, right? When you’re looking at a website or when I’m looking at a website, what I want to look to see is do I feel like they’re being transparent? Are they telling me everything? If I’m looking at the small print is this small print giving me information that seems positive or does it seem like it’s in small print because they didn’t want me to necessarily–?

David: If they’re trying to hide something.

Siobhan: Yes, right? Put it in bold print. That would be the first thing. Again, the use of the words “success rate,” what does that mean? If the success rate is defined in really tiny print, that could possibly be a red flag. You know? Tell me what you’re telling me, right? Is this the percent of cocaine addicts who have successfully completed a 30-day inpatient program? Okay, that’s a form of success, I completely get that. But tell me, cocaine, substance use disorder, cocaine.

David: What is it?

Siobhan: Right, exactly. Title the graph that for example, right? That would be the first thing. How transparent does it appear and then secondly, representation, right? What does it really represent? Is it representing a large percentage of the population that they’re treating or is it representing a very small? For example, testimonials, the films where the patient says how great this treatment was and how good their life is, those are beautiful.

I mean, I love, and who doesn’t love that. It speaks to your heart. You feel so good, you’re like “Yay, that can happen for me,” right? And everybody uses them. It’s an important part of it but is it just that person that’s representing what does a minuscule proportion of people who’ve been to your treatment center. Looking for both the testimonial but also something that represents, “We’ve gone to the trouble of tracking our patients to see how we’re doing because we want to make sure that we’re providing the service that and getting the results that people want.”

Something that is representative so looking to make sure that they’ve enrolled or included a lot of people in the research, they’re not cherry picking, they’re not only including some people in the research, and then, they’ve been able to follow up with a high percentage of those people.

David: Yes, that’s a good point because the sample size in a lot of ways is just as important as what type of patients and conditions are included in the data.

Siobhan: Exactly.

David: Speaking of that, is there a standard for a sample size that is good in order to be accurate?

Siobhan: Yes. SAMHSA’s numbers say that you need to enroll at least 80% of people in your evaluation for it to be reliable, to confirm that you’re not specifically excluding certain people. We enroll over 90% annually and have been since, say, 2013 or earlier. Then, they also say that you should be able to reach at least 60% of those people at the different time points after treatment. We have a team of people that make those phone calls and then send those emails and we’ve been able to reach an average better than 65% annually across those time points. It’s about achieving SAMHSA’s standards even if you’re not necessarily receiving SAMHSA’s funding.

David: For people who are looking at marketing from treatment centers, what are some other elements that you might direct them to look at?

Siobhan: I would also– Keeping in mind this question, for example, “How do you know?” When the site says, “We provide tool diagnosis treatment.” That’s awesome. How do you know? Are you checking to make sure? What audits are you doing? What standard are you using? For example, we hold ourselves to the standard of what’s called Treatment Improvement Protocol 42, TIP 42. It’s sort of the Bible of integrated treatment, right? We do an audit each year to make sure that we’re meeting those standards. What types of things like that? And then, what level of training do staff have? Is this master’s level trained staff, the therapists and clinicians that you’ll be seeing or is it maybe a master’s level person who serves four different sites provides oversight and everybody else is only certified? You want to be looking for licensed therapist. I would look at the level of staffing and training as well. Those are just some of the features and then, are they able to give you information in an aggregate form about their patients, even if it’s not meeting perfectly the standards, are they honest about that and are they able to tell you, “Well, we only reached 50% but here’s what we found.”

A lot of work went into that 50% and I get that. At least that effort is being made and there’s some competence to be had through that possibility. Those are some of the key factors that I would look for.

David: It’s kind of back to what you said about transparency. It’s about does it seem like they’re being open about whatever they have?

Siobhan: Exactly, and that they’re making an effort to be able to have that type of data, that it’s not just like, “Yes, we’ll talk to this guy, he said we did good.” If everything is dependent on a couple of quotes from five patients and a testimonial, I’m not saying those things are bad, but I’m saying the question then becomes what can you tell me about the overall population.

David: This is not really objective.

Siobhan: Exactly.

[music]

David: I know this can be difficult to define but how would you define success in your role, in your world of research?

Siobhan: For us, on an individual basis, success is a patient who first comes out of treatment saying, yes, they had an experience and they would recommend this treatment center to other people. That would be the first part of success because we know that a patient who is satisfied with their treatment process is much more engaged in their treatment processes. They’re going to continue on with whatever recommendations post-treatment there are and be more likely to experience positive long-term outcomes. In general, then, success, I guess the biggest indicator would be how is the quality of your life? Has the quality of your life overall improved?

Actually, that’s the next white paper that we’re going to be putting out. We’re looking at the improvement in just overall quality of life. For moms, success is like, “Wow, I can sit in the room with my teenage daughter again,” and to not be in a fight. Interestingly enough, when we asked the young adults, 18 to 25-year-olds, questions about their family relationships, they were asked, “In the past 30 days, how many days have you had serious conflicts with your mother?” Do you know how many days they responded at intake?

David: All of them.

Siobhan: Yes, 30. [laughs] 30 days.

David: I believe that.

Siobhan: Six months later it was down to eight.

David: That’s great.

Siobhan: Right? And so to me, I get chills with that, that’s just– “Yes. I still have serious conflict with my mom but–” and this is that percent, this is 90% of them were enrolled. We reached 65 or so percent of them after treatment. And that high percentage of people can now actually get along and maybe even go to lunch with mom without having a big fight. I think there’s so many different ways to define success. People think that numbers are cold and that they don’t tell the whole story but when you put it in those terms, I don’t think that’s cold at all. That really gives me great hope.

David: Yes, it’s a tangible representation of all the work that all these other departments have been putting in from the time that person calls in, the very first time, all the way through treatment.

Siobhan: Yes, absolutely all the way through the Life Challenge group. So it’s exciting. One of my other favorite statistics is we ask, “How troubled or bothered you are by your psych symptoms when you come in at intake and at different time points?” And, “In the past 30 days, how many days have you been troubled or bothered by your psych symptoms?” And that average answer is 21 days.

Over the past three years, it’s been anything 20 to 22. About 21 days. At the one year point, over the past three years, the average is eight. To me, that’s two weeks that someone got back in their life, two weeks that maybe they’ll make it to work today or they can cook breakfast for the family or take their children to the park and just be available for their life. That’s what treatment does for people.

[music]

David: You mentioned earlier that in addition to your research roles, you’re an Arise Interventionist and a master addictions counselor. Make sure I get that right. How would you say that having that experience on the “front lines” of addiction has helped shape your perspective with what you’re doing now?

Siobhan: It’s actually helped me really work with the sites with our facilities because I get this struggle of working at a facility, of what it’s like to be a clinician in this field on a day to day basis. It can be draining and it can be hard and it’s easy to get cynical and it’s easy to just develop the automatic eye roll. What I didn’t tell you, and I’m happy to share it, is I’m also in recovery. And the two combined, being in recovery and understanding how painful it can be to be out there, to be helpless.

I have family members who are in recovery and so to understand what it’s like to watch someone else make these mistakes and harm themselves and go through everything that we watch our loved ones go through and to feel helpless on that front and hopeless on that front. Having a master addiction counselor and being an Arise Interventionist as in not being able to fix it.

Then, to be a clinician, to be able to identify with other clinicians, speak their language, offer support and just, I think, knowing. It’s funny but the struggle is real, knowing what it’s really like to be in this field. Nobody goes into this field because they’re going to get so rich providing substance abuse counseling, right? They go into it because it’s a labor of love and to be able to try to, in the research, honor that and hold all three of those spaces as being why this matters. That is a big part of why we do what we do in the research team.

It’s truly to make a difference in the world, to be able to help people understand that not just one by one by one but as a group, recovery is possible. Things do get better. There is hope and to be able to share that hope with people in a meaningful and scientifically valid way it really means a lot to the entire team.

David: Yes, I think that’s all we have for today.

Siobhan: Thanks, that was awesome.

David: Thank you for dropping in and sharing that with us.

[music]

David: This has been the Recovery Unscripted podcast. Today we’ve heard from Siobhan Morse, the Division Director of Clinical Services for Foundations Recovery Network and UHS. If you’d like to read more about some of the research that Siobhan and her team have published, please visit foundationsrecoverynetwork.com. As always, thank you for listening. Please share this podcast, give us a rating on iTunes, and stay tuned for more episodes. See you next time.