This article reviews the lessons learned from a nonprofit’s chatbot project to dispense mental health advice that went wrong, and how nonprofits can reduce the risks when adopting ChatGPT and other generative AI (artificial intelligence) technologies.
Over the past five years of researching and writing about AI adoption for the nonprofit sector, we have argued for nonprofits to use AI ethically and responsibly—which means taking a human-centered, strategic, and reflective approach. It is the responsibility of senior management to plan the use of AI carefully in order to ensure that no harm is done to people inside and outside of the organization.
While researching and writing our book, The Smart Nonprofit: Staying Human Center in an Age of Automation, we found many examples of chatbots misbehaving in the world. While some incidents were entertaining, others were potentially damaging and taken offline. These included Microsoft’s Tay, Facebook’s Negotiation Bots, and Scatter Lab’s Luda.
Chatbots developed by nonprofits for programs and services have faced challenges at times as well. For example, DoNotPay, a legal services chatbot designed to challenge parking tickets and navigate small claims courts, faced criticism for potentially oversimplifying legal issues and providing inaccurate advice.
The recent release of ChatGPT, an incredibly potent example of the power of AI tools to mimic conversations and empathy, mental health providers have also been tempted to substitute bots for people. This is happening despite the fact that clinicians are still trying to figure out effective and safe use cases.
The demand for mental health hotlines skyrocketed during the pandemic and has continued ever since. At the same time, finding human counselors has been a struggle. Replacing staff or volunteer counselors with bots to save money has been viewed as a quick fix for some organizations.
The most recent example comes from the National Eating Disorders Association (NEDA), the largest nonprofit organization dedicated to supporting individuals and families affected by eating disorders. For over 20 years, its hotline has provided advice and help to hundreds of thousands of people experiencing eating disorders. Regardless of the technology (chat, phone, or text), a caller was always connected to a human being for advice. It is not unusual for hotline staff to talk to people who are suicidal, dealing with abuse, or experiencing some kind of medical emergency.
The affect of COVID-19 on the community of tens of millions of people worldwide who experience eating disorders was serious and profound and created a surge in hotline calls of over 100%. NEDA employees and approximately 200 volunteers could barely keep up with demand.
Not only were hotline and staff volunteers receiving many more calls, the calls, themselves, became increasingly serious in nature. People were often calling from situations of acute distress and showed signs of imminent and serious self-harm. Staff and volunteers lacked the training and resources to manage these calls, and began to feel burned out, requesting more support from leaders of the organization.
NEDA’s leadership looked at this situation, also noting the increasing risk profile for the nonprofit. If someone called in, spoke with a hotline volunteer, and then harmed themself seriously, could the organization be held legally responsible?
In response, in 2022 NEDA’s leadership unveiled Tessa, an AI-driven chatbot, and promptly handed out pink slips to hotline staff.
Replacing people this way is a dreadful idea on multiple levels.
First, a chatbot cannot understand the nuances of the pain hotline callers are in, nor can it provide the kind of empathy these callers need and deserve. This is not a hypothetical concern; one user reported trying the chatbot and stated, “[The chatbot] gave links and resources that were completely unrelated” to her questions.
A San Diego-based eating disorders consultant tested Tessa and received problematic responses from the chatbot encouraging her to lose weight and sharing diet tips. Needless to say, the advice given by Tessa runs contrary to treatment protocols accepted by medical experts in the field. Maxwell shared her experience on social media, stating: “If I had accessed this chatbot when I was in the throes of my eating disorder, I would NOT have gotten help for my ED. If I had not gotten help, I would not still be alive today.” The consultant brought enough attention to the matter for NEDA to shutdown Tessa on May 30th, just days before it had planned to completely phase out its telephone helpline, including all helpline staff.
Second, chatbots and other AI technologies need to be carefully designed and tested for human interaction and bias to mitigate risks. Even if a chatbot was a good idea for providing sensitive counseling support, it does not appear that this chatbot was well designed or adequately tested.
Third, AI technology should never be used primarily as a way for management to reduce headcount. As we wrote in The Smart Nonprofit, the goal of using AI-powered technology is to augment human efforts, not replace them. Co-botting is the sweet spot created when people do deeply human activities and bots perform rote tasks learned over millions of attempts with no harm.
In the above example, the NEDA team should have known about the potential danger of chatbots going off the rails. A simple query to ChatGPT about its potential dangers generated a half-dozen examples. Two of the examples it cited were made up, further reinforcing the idea that ChatGPT can give inaccurate information without proper training and supervision.
Take the do no harm pledge
Artificial intelligence used for mission-driven work must be held to the highest ethical and responsible standards. As the example of NEDA’s chatbot Tessa clearly shows, AI cannot and should not be considered an inexpensive replacement for human staff or volunteers. The benefits the technology offers can only be achieved when an organization embraces a human-centered, “do no harm” approach.
Here are the first steps for using AI in ways that do not cause harm:
- Stay human-centered: Responsible adoption requires informed, careful, strategic thought to ensure the technology is used to enhance our humanity and enable people to do the kinds of relational, empathetic, and problem-solving activities we do best. Responsible use also requires people to be in charge of the bots at all times.
- Lead with generative AI literacy: Increase your organization’s, including senior leadership’s, AI literacy. People in your organization do not need to know how to code, but they do need to understand what a chatbot is and isn’t capable of doing. In the NEDA example, the team should have known about the potential danger of chatbots going off the rails. A simple query to ChatGPT about its potential dangers generated a half-dozen examples. Two of the examples it cited were made up, further reinforcing the idea that ChatGPT can give inaccurate information without proper training and supervision.
- Think through co-botting use cases: AI should be used sparingly and carefully. This begins by identifying specific pain points that are bottlenecks for the organization. The next step is to outline exactly what tasks and decision-making people will retain as well as what tasks AI will do or what will be automated and requires human oversight to prevent harm. It is also important to solve the right problem. For example, The Trevor Project uses a Chatbot as a training simulator to train more volunteers to help scale demand for services. The bots never interact with people seeking out advice from the helpline, and the organization did not reduce headcount.
- Test, test, test: Generative AI is not a pot roast—you can’t set it and forget it. Chatbots need to be vigorously tested to mitigate risks. Humans need to check the answers the bots are providing for accuracy, bias, and hallucinations. They also need to put safeguards in place, especially if the chatbot is being used to provide advice for sensitive situations. The nonprofit and its technical partners also need to invest in ethical AI training, so they’re prepared to evaluate the chatbot’s interactions.
There are a lot of decisions to be made in order to use ChatGPT responsibly. There are also important mistakes to avoid. We will save you and your organization an enormous amount of time and pain by sharing this: It is important NOT to wait for something bad to happen before looking for warning signs of potential harm. Do your homework, think through the process carefully, take your time, and above all, do your best to reduce risks. If your nonprofit understands where threats may be lurking, ill-understood, or simply unidentified, you have a better chance of catching them before they catch up with your nonprofit.
Learn more about using technology to advance your mission by attending the Information Revolution – Managing Technology class on Saturday, July 22 from 1 – 5 pm as part of the Sunshine Certificate in Nonprofit Management.
Source: Candid blog. Candid has been a preferred partner since 2018 when Florida Nonprofits became an outlet for the Foundation Center Funding Network.