An AI That Can't Promise a Rescue
I built a phone line where an AI takes animal rescue reports. The interesting part isn't the voice pipeline. It's everything the model isn't allowed to do.
AI gathers information. Humans decide what happens next.
A volunteer animal charity I’m helping with has a problem that sounds simple and isn’t. Someone calls at half past two in the morning because there’s an injured fox on a fast road, or ducklings in a storm drain, or a cat nobody can reach. The volunteers are asleep, or out on another call, or simply not in a position to answer the phone well.
The obvious fix is an answering service. The less obvious fix is an AI that can actually hold a conversation, pull out the details a human volunteer needs, and stop short of pretending it can send help.
That’s what I built. A phone line where an AI answers, asks questions, and files a structured report. Humans still decide whether anyone goes out.
What happens when you call
Twilio receives the call and hits a Fastify webhook. The API validates the Twilio signature, returns TwiML pointing at a WebSocket, and opens a media stream. On the other side of that stream, a bridge connects to OpenAI’s Realtime API: audio in, audio out, server-side voice activity detection, no round-trip through a separate transcription step.
The model has one tool: create_incident. When it has enough information, it calls that tool with structured fields (species, situation, location, injury status, caller contact details). The tool validates the payload with Zod, runs it through a small emergency-policy module, writes a row to Postgres, and triggers an SMS alert to on-call volunteers. A Next.js dashboard lets humans review, assign, and add notes.
The whole thing is an npm workspaces monorepo: API, dashboard, shared types, Prisma schema, Docker Compose for local Postgres. Nothing exotic. The voice pipeline is maybe a hundred lines of WebSocket plumbing once you strip away the error handling.
The refusals are the product
The system prompt is explicit about what the AI is not:
- Not a veterinarian.
- Not an emergency dispatcher.
- Cannot promise a rescue.
- Cannot diagnose an animal.
- Cannot give treatment instructions.
At the start of every call, the assistant identifies itself as AI, explains that volunteers are busy, and says it cannot guarantee anyone will attend. If a caller asks what to do medically, it declines and suggests contacting a vet. If they ask for someone to come right now, it explains that volunteers review reports and it cannot promise a response time.
That sounds like a list of things you bolt on after the demo works. For this project, it was the design. A distressed caller will ask the voice on the phone to fix the problem. Letting the model improvise reassurance is how you end up with “someone is on their way” when nobody is.
I’d rather the AI sound slightly unhelpful than lie.
Location was harder than voice
The voice part was the easy bit. Twilio and OpenAI have docs. You wire up a WebSocket, pass audio/pcmu both ways, and it mostly works.
The hard part was location. Rural Wales has place names that sound alike on a bad line, postcodes that start with SA or SY or LD, and landmarks that mean something locally (“by the school”, “the old farm”) and almost nothing on a map. A wrong town sends a volunteer forty minutes in the wrong direction. That’s worse than getting the species wrong.
So the prompt spends more words on location than on anything else: nearest town first, then postcode if they have one, then a landmark. Repeat the place name back and ask the caller to confirm or spell it. If you’re not confident you heard it correctly, say what you heard and ask again, and record exactly what the caller said without inventing, translating, or “fixing” Welsh place names.
I learned this after watching the model confidently autocorrect a garbled village name into a different village that happened to sound similar. Helpful in a chatbot. Catastrophic when someone is standing in a ditch at midnight waiting for help that went to the wrong parish.
Keep the judgment in code
The model reports urgency as one of its fields. I do not trust that field on its own.
Between the tool call and the database write, everything passes through emergency-policy.ts: a small deterministic module with regex signal detection and a rank-based urgency ladder. Immediate danger bumps to immediate_danger. Serious injury language bumps to urgent. Trapped or stray patterns get tagged. If the caller asked for treatment advice, the policy attaches a standard vet-referral message. Every incident gets flagged for human review regardless.
There are unit tests for this. Tests for the voice flow, the incident tool validation, auth, retention. The parts where a wrong answer has consequences are plain TypeScript, not prompt engineering.
I’m not against prompts. The intake assistant needs a personality and a conversation flow. But “is this animal in immediate danger?” is not a question I want answered by vibes.
The boring stuff matters more here
Phase six of the project was not voice quality. It was API keys on every incident route, rate limiting, Helmet security headers, Twilio webhook signature validation, HTTPS enforcement behind a reverse proxy, caller phone numbers redacted on list views, call recording disabled by default, and a retention scheduler that purges incidents older than a year.
This system collects real phone numbers and real locations from people who are stressed. That’s a different threat model from a side-project todo app. I don’t want call recordings sitting on a disk unless someone explicitly opts in. I don’t want incident lists showing full phone numbers to anyone with dashboard access. I don’t want data hanging around forever because I forgot to write a purge job.
None of that is interesting to demo. All of it is necessary before I’d point a real phone number at the thing.
What it doesn’t do
This is an intake line, not a dispatch system. The AI does not decide who goes out, does not route to the nearest volunteer, does not integrate with vet systems, and does not replace the human triage step where someone looks at the report and thinks “can we actually help this, tonight, with what we have?”
It gathers information when humans can’t answer the phone. That’s a narrow job, and I think that’s the right scope for an MVP.
The voice still breaks in ways I haven’t fully solved. OpenAI Realtime cold starts add latency on the first response, Welsh place names on a mobile connection in the rain are still hard, and the model occasionally asks a question the caller already answered despite instructions not to. Volunteers still need to check the dashboard, not just read the SMS.
But at two in the morning, a structured report with a confirmed location and a callback number is better than voicemail nobody checks until breakfast.
If you’re building something similar: spend more time on what the AI must refuse to do than on making it sound clever. The voice is the interface. The constraints are the product.