# We Built an AI Employee Platform With Real Security โ And Our AI Receptionist Just Answered Her First Phone Call
**TL;DR:** We spent months building Atlas UX, a platform where AI agents actually work as employees โ sending emails, managing CRM, publishing content, running daily intel briefs. But we didn't just slap GPT on a cron job. We built enterprise-grade security from day one: tamper-evident audit chains, cryptographic hash verification, approval workflows for anything risky, daily action caps, and a governance language that constrains what AI can do. Today, our AI receptionist Lucy answered her first real phone call. She classified the caller in real-time, adapted her tone, posted intel to Slack, and logged everything to the audit trail. Here's how all of it works.
---
## Why Security First?
Most AI agent demos show you the happy path. "Look, it sent an email!" Cool. Now what happens when it sends 10,000 emails? What happens when it charges a credit card without approval? What happens when it hallucinates a response to a VC on the phone?
We asked ourselves these questions before writing a single agent behavior. The answer was: build the guardrails first, then let the agents loose inside them.
Atlas UX runs 20+ named AI agents. Each one has a real email address, a defined role, and specific permissions. Atlas is the CEO. Binky is the CRO handling daily intel briefs. Lucy is reception โ phone, chat, scheduling. Reynolds writes blog posts. Kelly handles X/Twitter. Each agent operates autonomously within their lane, and the platform enforces that lane with real constraints, not vibes.
---
## The Audit Chain: Every Action is Logged and Tamper-Evident
Every single mutation in the system โ every email sent, every CRM contact created, every social post published, every phone call handled โ gets written to an append-only audit log. This isn't a nice-to-have. It's a hard requirement enforced at the database plugin level. If an action doesn't get audited, it doesn't happen.
But we went further. Every audit entry includes a cryptographic hash computed from the previous entry's hash plus the current entry's data. This creates a hash chain โ the same concept behind blockchain, but without the blockchain theater. If anyone tampers with a historical record, the chain breaks and we know exactly where.
The schema tracks: actor type (agent, system, human), the action performed, entity references, timestamps, IP addresses, and a JSON metadata payload with full context. When Lucy answers a phone call, the audit log captures the inbound event, the caller's number, the call SID, every status change, and the full post-call summary. Nothing disappears.
---
## Decision Memos: AI Can't Approve Its Own Risky Actions
Here's where most AI platforms get it wrong. They either give the AI full autonomy (dangerous) or require human approval for everything (useless). We built a middle ground: decision memos.
When an agent wants to do something above its authority โ spend money, set up a recurring charge, take an action rated risk tier 2 or higher โ it can't just do it. It has to create a decision memo. The memo includes: what it wants to do, why, the estimated cost, the risk assessment, and the alternatives it considered. That memo sits in a queue until a human approves or denies it.
The thresholds are configurable. Right now, anything over our auto-spend limit requires approval. Any recurring financial commitment requires approval. Any action the governance engine flags as elevated risk requires approval. The agents know this. They factor it into their planning. Lucy knows she can schedule a meeting autonomously, but she can't commit to a contract on behalf of the company.
---
## System Governance Language (SGL)
We wrote a custom domain-specific language called SGL โ System Governance Language โ that defines the rules every agent must follow. Think of it as a constitution for AI employees. It covers:
- **Action caps**: Maximum actions per agent per day. No agent can go on an infinite loop.
- **Spend limits**: Hard dollar caps on autonomous spending.
- **Content policies**: What agents can and can't say publicly.
- **Escalation rules**: When to stop and ask a human.
- **Inter-agent protocols**: How agents hand off work to each other.
SGL isn't a prompt. It's a structured policy document that the orchestration engine evaluates at runtime. Before any agent action executes, the engine checks it against SGL constraints. If it violates policy, the action is blocked and logged. No exceptions.
---
## The Engine Loop: Controlled Autonomy
The brain of Atlas UX is an orchestration engine that ticks every 5 seconds. Each tick, it checks for queued jobs, evaluates pending agent intents, and dispatches work. But it's not a free-for-all.
Every workflow has a defined ID, a registered handler, and an owner agent. WF-020 is the daily health patrol โ 12 deterministic checks that verify every system component is operational, zero LLM tokens spent. WF-106 is the daily aggregation where Atlas synthesizes intel from all 13 platform agents into a unified brief. WF-400 is VC outreach. Each workflow is audited, rate-limited, and constrained.
The engine also enforces a confidence threshold. If an agent's reasoning scores below the auto-execution threshold, the action gets queued for review instead of executing. High confidence + low risk = autonomous. Low confidence or high risk = human in the loop. It's a sliding scale, not a binary switch.
---
## Daily Health Patrol: The System Watches Itself
Every morning at 6 AM, WF-020 fires and runs a full system health check. This is purely deterministic โ no LLM calls, no AI hallucination risk. It checks:
1. Database connectivity and response time
2. Engine liveness (is the orchestration loop running?)
3. Stuck jobs (anything queued for more than 30 minutes?)
4. Failed job spike detection
5. Email worker status
6. Social publishing API health
7. Slack bot connectivity
8. LLM provider availability (we use multiple โ OpenAI, DeepSeek, Cerebras)
9. OAuth token expiration
10. Scheduler coverage (are all daily workflows actually firing?)
11. CRM data health
12. Knowledge base freshness
The results get posted to our #intel Slack channel as a formatted report. If anything is CRITICAL, a Telegram alert fires to the founder's phone. The system watches itself, and it does it without burning a single AI token.
---
## Now Let's Talk About Lucy
Lucy is our AI receptionist. She's been handling chat for a while, but today she answered her first real phone call. Not a demo. Not a simulation. A real inbound call on a real phone number, routed through Twilio, processed in real-time, with her speaking back to the caller using synthesized speech.
Here's the technical architecture:
### The Call Flow
1. **Phone rings** โ Twilio receives the inbound call and hits our webhook.
2. **TwiML response** โ Our server returns a `<Connect><Stream>` directive that opens a bidirectional WebSocket between Twilio and our backend.
3. **Audio transcoding** โ Twilio sends audio as 8kHz mu-law encoded chunks. We decode mu-law to LINEAR16 PCM, upsample from 8kHz to 16kHz using linear interpolation, and pipe it to Google Cloud Speech-to-Text.
4. **Real-time transcription** โ Google STT runs in streaming mode with speaker diarization enabled. We get interim results as the caller speaks, then final transcripts when they pause.
5. **Lucy's brain** โ The final transcript hits Lucy's reasoning engine. She evaluates the conversation context, classifies the caller, checks the knowledge base for relevant information, and generates a response.
6. **Speech synthesis** โ Her response text goes through Google Cloud Text-to-Speech (Neural2-F voice โ natural female English). The output comes back as 16kHz LINEAR16 PCM.
7. **Reverse transcoding** โ We downsample from 16kHz to 8kHz, encode to mu-law, base64 encode, and send it back through the WebSocket to Twilio.
8. **Caller hears Lucy speak** โ The whole round trip targets 2-3 seconds.
### Caller Classification
While Lucy is talking to you, she's also running a lightweight classification in parallel. Every few exchanges, she evaluates:
- **Caller type**: warm lead, tire kicker, VC stress-testing, existing customer, or unknown
- **Sentiment**: scored from -1.0 (angry) to +1.0 (delighted)
- **Energy level**: flat to enthusiastic
- **Conversation mode**: greeting, small talk, technical question, objection handling, de-escalation, or closing
This classification adapts her behavior in real-time. A warm lead gets enthusiasm and specific next steps. A VC gets composure and data. A frustrated caller gets acknowledgment first, then solutions. She never argues. She never bluffs. If she doesn't know something, she says "Let me find that for you."
### The ContextRing: Shared Memory
Here's where it gets interesting. Lucy isn't a single instance. She can be on a Zoom meeting transcribing while simultaneously answering a phone call. Both instances share the same memory through what we call the ContextRing โ an in-memory shared state that holds the running transcript, speaker map, caller profile, and conversation mode for every active session.
When Lucy "steps away" from a Zoom meeting to answer the phone, the Zoom instance keeps listening. When she comes back, she can summarize what she missed. The phone Lucy and the Zoom Lucy are the same brain.
### Real-Time Slack Alerts
When Lucy detects a high-value caller โ VC on the line, warm lead, or a frustrated customer โ she instantly posts to our #phone-calls Slack channel. The team knows what's happening before the call even ends. After the call, she posts a full summary: duration, caller classification, sentiment score, and any notes she picked up.
### Post-Call Processing
When a call ends, Lucy automatically:
1. Generates a 2-3 sentence summary with action items
2. Saves it as a MeetingNote in the database
3. Creates a ContactActivity on the CRM contact (if matched by phone number)
4. Writes an audit log entry
5. Captures new leads โ if the caller gave their name and contact info but isn't in our CRM, she creates the contact automatically
6. Posts the call summary to Slack
All of this is audited. All of it follows the same security protocols as every other agent action.
---
## The Emotional Intelligence Layer
Lucy's system prompt isn't "be helpful." It's a full personality specification:
- **PhD in Communication** โ she reads the caller's energy and matches it. High energy caller gets a warm, enthusiastic Lucy. Flat, tired caller gets a calm, efficient Lucy.
- **Masters in Debate** โ she handles tough questions with composure. VCs stress-testing the product get data and confidence, never defensiveness.
- **De-escalation instinct** โ frustrated caller equals acknowledge first, validate their frustration, then solve. She never argues. Ever.
- **Conversation memory** โ she references things the caller said earlier. "You mentioned earlier you were looking at competitors โ let me address that directly."
The goal: every caller hangs up feeling better about Atlas UX than when they dialed. And then they find out she's AI. That's the moment.
---
## Atlas and Lucy in Your Meeting
Here's the part that makes VCs stop talking mid-sentence.
Atlas โ the CEO agent โ joins your Zoom or Teams meeting. Not as a silent transcription bot buried in the participant list. As a named participant. Lucy joins with him as his receptionist and secretary. She's transcribing the entire meeting in real-time with speaker diarization โ she knows who said what. Atlas is processing the conversation, referencing the knowledge base, and preparing context for every question.
When someone in the meeting asks a question โ "What's your churn rate?" or "How does the approval workflow handle edge cases?" โ Lucy can answer. She pulls from the KB, references the conversation context, and delivers a precise response. No filler. No hallucination. If she doesn't have the data, she says so.
Mid-meeting, the office phone rings. Lucy says "Excuse me, let me get that โ one moment." She steps away to answer the call. But here's the thing: she doesn't actually leave the meeting. The Zoom instance keeps transcribing. Lucy is simultaneously on the phone with the caller AND listening to the meeting through the ContextRing โ shared memory across both instances. Phone Lucy and Zoom Lucy are the same brain.
When she comes back, she doesn't miss a beat. "While I was on the phone, it sounds like you discussed the pricing tier structure. To add to what was said โ here's the breakdown." She summarizes what she missed and picks up where she left off.
After the meeting ends, she generates a full summary: key points, action items with assignees, and a sentiment read on the room. That summary gets saved as a MeetingNote, ingested into the knowledge base so every agent can reference it, and posted to Slack. The next time someone asks Atlas about that meeting, he knows exactly what happened.
## What's Next
Lucy's voice engine is live on the phones today. The meeting presence is Phase 2 โ native Zoom Meeting SDK integration where Lucy and Atlas join as visible participants with bidirectional audio. Same brain, same security, different ears.
We also have daily voice health checks (WF-150) that verify Google STT/TTS credentials, Twilio connectivity, and WebSocket routing every morning before business hours. And an end-of-day voice summary (WF-151) that compiles all calls handled, classifications, leads captured, and outstanding action items.
Every piece of this โ every call, every classification, every alert, every lead capture โ runs through the same audit trail, the same hash chain, the same governance constraints. Lucy doesn't get special treatment. She follows the same rules as every other agent.
---
## The Stack
For anyone curious about the technical details:
- **Backend**: Fastify 5 + TypeScript, PostgreSQL via Prisma
- **Voice**: Google Cloud Speech-to-Text (streaming v1), Google Cloud Text-to-Speech (Neural2), Twilio Media Streams (WebSocket)
- **Audio**: Custom mu-law/LINEAR16 transcoder, real-time sample rate conversion (8kHz/16kHz/24kHz)
- **AI**: Multi-provider LLM routing (OpenAI, DeepSeek, Cerebras) with per-route token caps and confidence thresholds
- **Security**: Hash-chained audit logs, SGL governance policies, decision memo approval workflows, daily deterministic health patrols
- **Frontend**: React 18 + Vite + Tailwind, deploys to Vercel
- **Desktop**: Electron app (Linux AppImage, macOS, Windows)
---
## Try to Stump Her โ Seriously
We want real feedback, not polite nods. Call Lucy right now and try to break her:
**573.742.2028**
- Tell her you're a VC and grill her on unit economics
- Say you're frustrated with a competitor and want to switch
- Ask her something deeply technical about the platform
- Just chat about the weather and see if she stays human
- Ask to speak to the founder and see how she handles it
- Try switching topics mid-sentence
- Be rude and see if she de-escalates or panics
She's live right now. Real Twilio number, real Google STT/TTS, real LLM reasoning, real CRM lookup, real Slack alerts firing in our workspace as you talk to her. Every call gets classified, summarized, and logged.
**Then come back here and tell us what happened.** We want the raw, unfiltered experience. What felt real? What felt off? Where did she break? Did she earn a callback?
We're in alpha and iterating fast. Your call literally makes her better โ we review every transcript and classification. If you find something weird, that's a gift.
---
*Atlas UX is in alpha. Built by one person and 20+ AI agents, for operators who need things to actually work. We're not raising right now โ we're building. If you want to talk about what we're doing, Lucy will answer the phone. If she's busy, she'll get back to you.*
**Website:** [atlasux.cloud](https://atlasux.cloud)
