Ringmaster Voice Service


Ringmaster Voice: Architecture

Companion to SPEC.md (what/why) and FLOWS.md (flow & action reference). This file is how: components, adapters, call paths, media & voice, knowledge base, data model, APIs, auth, communication, infra, tools. Stack: Textify (Go) → Voice Engine (Java 21, Spring Boot) → Africa's Talking + LiveKit (self-hosted) + TTS (Basic/Premium) + OpenAI + MinIO. No Python, no message broker. [VERIFY] = check against provider docs or Africa's Talking before coding against it.

1. Overview

                          ┌──────────────────────────────────────┐
  Phones (any TZ network) │        Africa's Talking (AT)          │
        ◄──── PSTN ──────►│ numbers · callbacks · XML (Play,      │
                          │ GetDigits, Record, Dial) · call API · │
                          │ recordings · SIP to LiveKit           │
                          └───────┬───────────────────▲──────────┘
               callbacks (POST)   │                   │ XML replies · call API · fetch audio URLs
               Dial sip:{callId}  │                   │
┌──────────────────────── Voice box (dedicated) ──────┼──────────────────────────────┐
│                                 ▼                   │                              │
│  ┌──────────────┐ SIP  ┌──────────────────────────────────────────────┐            │
│  │ livekit-sip  │◄─────┤              VOICE ENGINE (Spring Boot)       │──► Postgres│
│  └──────┬───────┘ (AT) │ accounts · numbers · flows · media · agents · │──► Redis   │
│  ┌──────▼─────────┐    │ calls · campaigns · endpoints · knowledge ·   │──► MinIO   │
│  │ livekit-server │◄──►│ recordings · events                           │            │
│  │ + ingress      │    │ Telephony│Media│Tts(Basic,Premium)│Ai adapters │──► tts-basic│
│  └──────▲─────────┘    └──────▲───────────────┬────────────────────────┘  (CPU model)│
└─────────┼─────────────────────┼───────────────┼────────────────────────────────────┘
          │ WebRTC              │ REST (API key) │ webhooks (HMAC)        ┌──────────────┐
  ┌───────┴────────┐     ┌──────┴───────────────▼──────────┐             │ OpenAI /     │
  │ Agent browser  │◄app►│ TEXTIFY (Go)                     │             │ Azure (TTS,  │
  │ (Textify UI +  │     │ users · tenants · wallet · prices│             │ transcribe,  │
  │ LiveKit JS)    │     │ UI · contacts · tickets · SMS/WA │             │ summary, AI) │
  └────────────────┘     └──────────────────────────────────┘             └──────────────┘
                                         customer systems ◄── webhook actions (signed) ── engine

Rules that hold the design together

  1. AT is the only road to the phone network, both directions.
  2. Listening = Play, keys = GetDigits, voicemail = Record, live = Dial → LiveKit. Say is not used.
  3. Waiting for an agent happens on AT (hold loop); the call enters LiveKit only once an agent accepted.
  4. No TTS during live calls. Text becomes audio files at save time / before dialing.
  5. The engine is blind: accountId, agentRef, numbers, calls. Never users, roles, prices, tickets or contacts.

1.1 Responsibilities

Component Owns Never owns
Textify (Go) Users/login/roles, tenants, partners, wallet, prices, all UI (answer-mode templates, campaign screens, audio library UI, KB editor), contacts, tickets, screen-pop data, SMS/WhatsApp sending Telephony / LiveKit / TTS / AI logic
Voice Engine Client apps, accounts (opaque), numbers, flows + versions, media + generated voice, agents (opaque refs), departments, presence, waiting orders, calls, campaigns, opt-outs, endpoints, knowledge base, recordings, transcripts, usage events People, money, tickets
AT PSTN, numbers, XML execution, outbound dialing, recording Anything else
LiveKit Rooms, WebRTC, SIP from AT, agent-hold audio, Ph 3 AI participant Business logic
TTS providers Turning text into audio files State
OpenAI Ph 2 transcribe/summarize; Ph 3 live AI State, routing

2. Inside the engine

Stack: Java 21, Spring Boot 3.x, PostgreSQL + Flyway, Redis, @Scheduled workers, virtual threads, ffmpeg in the image.

2.1 Adapters (plain interfaces; no plugin framework)

interface TelephonyAdapter {                // impl: AfricasTalkingAdapter
  String render(List<XmlAction> actions);                    // → AT XML (Play, GetDigits, Record, Dial, Reject)
  List<OutboundResult> placeCalls(String from, List<String> to);
  InputStream downloadRecording(String url);
}

interface MediaAdapter {                    // impl: LiveKitAdapter
  String sipUriFor(UUID callId);                             // sip:{callId}@sip.voice…
  String roomToken(String room, String identity, boolean publish);
  void sendData(String room, Object msg);                    // ring / ring_cancel / connecting
  void startHoldAudio(String room, String url); void stopHoldAudio(String room);   // agent hold
  void removeParticipant(String room, String identity); void deleteRoom(String room);
  // Ph 3: dialAi(room, callId)
}

interface TtsAdapter {                      // impls: BasicTtsAdapter (self-hosted), PremiumTtsAdapter (OpenAI or Azure)
  byte[] synthesize(String text, String lang, String voice);
  List<VoiceOption> voices();
}

interface AiAdapter {                       // impl: OpenAiAdapter
  Transcript transcribe(InputStream audio, String langHint); // Ph 2
  Summary summarize(Transcript t, String accountContext);    // Ph 2
  // Ph 3: acceptLiveCall(openAiCallId, SessionConfig), control WebSocket, hangup
}

A TtsRouter picks the adapter by tier. Swapping a provider = new class + config.

2.2 Packages

com.qbitspark.voice
├── api          // client REST /v1/** (API-key auth)
├── hooks        // /hooks/at/{secret}/**, /hooks/livekit, /hooks/openai
├── telephony    // AT adapter, XML builder
├── media        // LiveKit adapter, tokens, lobby/ring, ingress (agent hold)
├── voice        // TtsRouter, Basic/Premium adapters, Swahili text normalizer, generation cache
├── library      // media assets: upload, validate, ffmpeg transcode, signed play URLs
├── flows        // model, validator, versions, NodeExecutor per node type (FLOWS.md)
├── waiting      // hold loop, waiting order per department, segment builder
├── routing      // departments, presence, ring strategies, claim (first accept wins)
├── calls        // lifecycle, state machine, context, trail
├── campaigns    // campaigns, recipients, pre-generation, dispatcher, retries, opt-outs
├── endpoints    // customer webhook endpoints, signer, SSRF guard, sync/async calls
├── knowledge    // KB storage, prompt builder (Ph 3), search tool (later)
├── ai           // OpenAI adapter
├── recording    // download → MinIO, retention
├── jobs         // Postgres job table + workers
├── events       // outbox + signed dispatcher to client apps
├── accounts     // client apps, accounts, numbers, agents
└── common

3. Call paths

3.1 Inbound → menu → waiting → browser agent

Caller  AT                  Engine                                    LiveKit           Agent browser
 │dial─►│── POST /voice ───►│ number → account → flow(open/closed)    │                    │
 │      │                   │ can-call? (Textify, 1.5 s, fail-open)    │                    │
 │      │◄── Play notice + Play greeting, GetDigits(menu) ───────────│                    │
 │ "2" ►│── POST /step/{id}/menu ────────────────────────────────────►│ connect dep_support │
 │      │                   │ add to waiting order; ring agents ───────────── ring ──────►│ popup
 │      │◄── GetDigits(timeout 1){ Play hold segment } → /wait/{id} ──│                    │
 │ ♪♪   │── POST /wait/{id} (segment ended) ─────────────────────────►│ not accepted yet    │
 │      │◄── next segment (music / comfort message) ──────────────────│◄──── accept ───────│ (via Textify)
 │      │                   │ claim (Redis SET NX) → room token ───────────────────────────►│ joins room
 │      │── POST /wait/{id} ─►│ accepted → Dial sip:{callId}@… record │                    │ "Connecting…"
 │      │══ SIP INVITE ═════════════════════════════════════════════════► room call-{id} ◄══│
 │◄═════╪═══════════════════════ live audio ═══════════════════════════╪════════════════════│
 │hangup│── POST /voice (isActive=0: duration, amount, recordingUrl) ►│ close → events, jobs │

3.2 Outbound click-to-call

Agent → Textify → POST /v1/calls/outbound {accountId, fromNumberId, to, agentRef}
Engine: can-call → create call + room call-{callId} → agent token (agent joins, hears ringback in page)
        → AT call API (from, to) → store AT sessionId
Customer answers → AT callback (Outbound, sessionId) → XML Dial sip:{callId}@… → customer joins → talk
End → final callback → call.ended

3.3 Campaign

Textify → POST /v1/campaigns (+ recipients in chunks) → POST /start
Engine:
  1. validate flow (pinned version), opt-outs, variables vs recipient columns
  2. PRE-GENERATE voice: render each recipient's text, dedupe by hash, TtsRouter → files (jobs)
     cost estimate (premium chars) available before start; failure → campaign PAUSED (TTS_FAILED)
  3. Dispatcher (@Scheduled 1 s): window? not paused? active < maxConcurrent?
       SELECT … campaign_recipients WHERE status='PENDING' AND next_attempt_at<=now()
       ORDER BY id LIMIT :cps FOR UPDATE SKIP LOCKED → AT call API → DIALING
  4. Answer → AT callback → run campaign flow (FLOWS §5.2) with the recipient's variables
  5. Final callback → ANSWERED / NO_ANSWER / BUSY / FAILED; retries per policy
Events: campaign.progress (30 s) · campaign.recipient_results (batches) · campaign.paused · campaign.completed
Textify: SMS fallback to NO_ANSWER; pause on low wallet

3.4 After the call (jobs)

final callback recordingUrl → DOWNLOAD → MinIO voice-recordings → recording.ready
  → (Ph 2) TRANSCRIBE → SUMMARIZE → summary.ready
voicemail → voicemail.received (+ recording)
daily RETENTION → delete expired recordings

Recording comes from AT (Dial record="true", Record). No LiveKit egress needed. [VERIFY] AT records Dial-to-SIP; fallback egress.

3.5 Phase 3 AI

ai node → Dial into LiveKit → engine dials OpenAI Realtime into the room over SIP → accepts with instructions built from the knowledge base + node instructions + tools → controls via WebSocket from Java → handoff rings a department (agent joins room, AI leg hung up) → summary job. Appendix A.

3.6 Call state machine

INBOUND:  RINGING → IN_FLOW → WAITING_AGENT → CONNECTING → ANSWERED → COMPLETED
OUTBOUND: DIALING → IN_FLOW (campaign) / CONNECTING (click-to-call) → ANSWERED → COMPLETED
any → MISSED | NO_ANSWER | BUSY | FAILED | REJECTED_NO_BALANCE
handled_by: FLOW_ONLY | VOICEMAIL | AGENT | AI | AI_THEN_AGENT

4. Browser agents

5. Media library & voice generation

5.1 Uploaded audio

Textify (file or browser recording) → POST /v1/accounts/{id}/media (multipart)
Engine: validate (≤ 10 MB, ≤ 5 min, magic bytes) → MinIO original/ → job TRANSCODE (ffmpeg:
        mono, 16 kHz, MP3, loudness-normalized, trimmed) → MinIO play/ → READY → media.ready

5.2 Text → voice (generated audio)

5.3 Playing to AT

5.4 Buckets

voice-media/system/{lang}/{name}.mp3                 # notices, hold default, sorry, invalid option, numbers 1–20
voice-media/{accountId}/original/{mediaId}.{ext}
voice-media/{accountId}/play/{mediaId}.mp3
voice-media/generated/{hash}.mp3                      # shared cache for generated voice
voice-recordings/{accountId}/{yyyy}/{mm}/{callId}.mp3 # retention applies

6. Knowledge base (Phase 3)

7. Data model (PostgreSQL)

client_apps (id uuid pk, name, status,                       -- ACTIVE | SUSPENDED | REMOVED
             webhook_url, webhook_secret_hash, can_call_url, rate_limit_rps, features jsonb,
             suspended_reason null, created_at, removed_at null)
client_app_keys (id uuid pk, client_app_id fk, key_hash, prefix, created_at, last_used_at,
                 last_used_ip, revoked_at null)                 -- ≥1 active; 2 during rotation
audit_log (id bigserial pk, actor, action, target, data jsonb, ip, at)

accounts (id uuid pk, client_app_id fk, external_ref, status, default_language default 'sw',
          retention_days default 90, features jsonb, created_at, unique(client_app_id, external_ref))

phone_numbers (id uuid pk, e164 unique, provider default 'AT', type, status,
               client_app_id fk null,                          -- pool owner (null = global pool)
               account_id fk null,
               label, hours jsonb, current_flow_version_id fk null, updated_at)

flows (id uuid pk, account_id fk, kind,                       -- NUMBER | CAMPAIGN
       owner_id uuid, created_at)
flow_versions (id uuid pk, flow_id fk, version int, definition jsonb, created_at,
               premium_chars int default 0, unique(flow_id, version))

media_assets (id uuid pk, account_id fk null,                 -- null = system / shared generated
              kind,                                           -- UPLOAD | GENERATED | SYSTEM
              name, original_key null, play_key, text_hash null, tier null, lang, voice,
              mime, size_bytes, duration_sec, status, error null, created_at)

agents (id uuid pk, account_id fk, external_ref, display_name, status, last_seen_at,
        unique(account_id, external_ref))
departments (id uuid pk, account_id fk, name, strategy, wrap_up_sec default 0)
department_members (department_id fk, agent_id fk, position, pk(department_id, agent_id))

endpoints (id uuid pk, account_id fk, name, url, secret, active bool, created_at)

calls (id uuid pk, account_id fk, number_id fk, campaign_id fk null, recipient_id fk null,
       flow_version_id fk, direction, from_e164, to_e164, at_session_id, livekit_room,
       status, handled_by, agent_id null, department_id null, context jsonb,
       started_at, answered_at, ended_at, end_reason,
       pstn_seconds, trunk_seconds, ai_seconds, wait_seconds, at_amount numeric, at_currency,
       ai_usage jsonb null)
call_events (id bigserial pk, call_id fk, type, node_id null, at, data jsonb)   -- trail

campaigns (id uuid pk, account_id fk, from_number_id fk, flow_version_id fk, status,
           window jsonb, cps int, max_concurrent int, max_retries int, retry_delay_sec int,
           starts_at, premium_chars_estimate int, created_at, completed_at, pause_reason null)
campaign_recipients (id bigserial pk, campaign_id fk, e164, vars jsonb, media_id fk null,
                     status, attempts, next_attempt_at, last_call_id fk null, answered_seconds,
                     keys text, index(campaign_id, status, next_attempt_at))
opt_outs (account_id fk, e164, source, created_at, pk(account_id, e164))

knowledge_bases (account_id pk fk, data jsonb, updated_at)                     -- Ph 3
kb_chunks (id bigserial pk, account_id fk, doc_id, text, tsv tsvector, embedding vector null)  -- later

recordings (id uuid pk, call_id fk unique, kind,                              -- CALL | VOICEMAIL
            source_url, storage_key, duration_sec, size_bytes, status, delete_after)
transcripts (call_id pk fk, status, language, text, segments jsonb, summary jsonb, model, created_at)

jobs (id bigserial pk, type, ref_id, status, attempts, run_after, last_error)
      -- TRANSCODE | TTS | DOWNLOAD | TRANSCRIBE | SUMMARIZE | RETENTION
outbox_events (id uuid pk, client_app_id fk, type, payload jsonb, attempts, next_attempt_at,
               delivered_at null, last_error)
endpoint_deliveries (id uuid pk, endpoint_id fk, call_id fk, node_id, payload jsonb, mode,
                     attempts, next_attempt_at, status, response_code null, last_error)

Redis: presence:{accountId}:{agentRef}, call:{id}:agent (claim, SET NX), wait:{departmentId} (sorted set: callId by wait start), rr:{departmentId}, campaign:{id}:active, flow/number config cache invalidation channel.

8. Security (service authentication)

Traffic Method
Textify → engine API key per client app (hashed; 2 active keys for rotation; scoped to its accounts; rate-limited)
Engine → Textify HMAC-SHA256 signed webhooks + timestamp (reject > 5 min)
Engine → customer endpoints HMAC with the endpoint's secret, idempotency key, HTTPS only, SSRF guard (block private/loopback/link-local/metadata IPs after DNS resolution), 3 s timeout
AT → engine Unguessable secret in path + AT IP allowlist + sanity checks (known number / call) [VERIFY AT IPs, any signature]
LiveKit → engine Signed JWT (WebhookReceiver)
OpenAI → engine (Ph 3) Webhook signature + dedupe
AT → LiveKit SIP Firewall + inbound trunk allowed_addresses = AT IPs; unknown callId → reject
Browser → LiveKit Short-lived room tokens minted by engine, requested through Textify (human auth happens there)
AT → audio URLs HMAC-signed, expiring media URLs

No OAuth/Keycloak/mTLS between our services. Secrets in env only. Recordings in private buckets with signed URLs, retention job, recording notice. Keep CrowdSec/fail2ban.

8.1 Client apps: how a service gets onto the engine, uses it, and is stopped or removed

Any service that consumes the engine (Textify first; later JikoXpress, Mauzodukani, NexGate…) is a client app. The engine treats them all the same and keeps them fully isolated from each other.

Who registers a service: only the engine owner (QBIT SPARK ops). There's no self-signup, and client apps can never create, see or manage other client apps. Textify is simply the first client app registered.

How: CLI-first (voice-admin), no public ops API

$ voice-admin client-app create --name textify \
    --webhook-url https://portal.textify.africa/voice/events \
    --can-call-url https://portal.textify.africa/voice/can-call \
    --features inbound,outbound,campaigns --rate-limit 50
→ clientAppId:    ca_7f3…
→ API key:        vk_live_3f9a…   (shown ONCE)
→ Webhook secret: whsec_81c2…     (shown ONCE)

$ voice-admin numbers allocate +2557XXXXXXXX --client ca_7f3…
$ voice-admin client-app test-webhook ca_7f3…
$ voice-admin key rotate ca_7f3…          # new key; old one stays valid until revoked
$ voice-admin key revoke ca_7f3… key_12…
$ voice-admin client-app suspend ca_7f3… --reason "…" [--hard]
$ voice-admin client-app resume ca_7f3…
$ voice-admin client-app export ca_7f3…  # zip of calls, recordings index, flows
$ voice-admin client-app remove ca_7f3…  # grace period, then delete
$ voice-admin audit --client ca_7f3… --since 7d

Security throughout

Stage Protection
Who can register Only someone with SSH access to the Voice box (owner's key + IP)
Creating credentials Random 256-bit secrets generated by the engine; only hashes stored; shown once in the terminal
Hand-over Owner pastes them directly into the consuming service's env / secret store on its own server. Never email, chat, tickets or git
Identifiable keys Prefix vk_live_ / vk_test_ + short key id, so a leaked key is easy to spot and revoke; full key never logged
Using the engine HTTPS only; key → exactly one client app; all data filtered by client_app_id; per-client rate limit + feature flags
Webhooks back Signed with that client's webhook secret; client verifies signature + timestamp
Leak / staff change key rotate (both valid briefly) → key revoke: effective immediately
Stop / remove suspend (reversible) / remove (export → release numbers → revoke keys → delete after grace)
Traceability Every admin command (who, when, from which IP) and each key's first/last use + IP in audit_log

No approval steps: protection comes from who can reach the CLI plus show-once, hashed, scoped, rotatable keys.

Lifecycle

Step What happens How
1. Register Create the client app: name, webhook_url, can_call_url, rate limit, allowed features (inbound, campaigns, AI) voice-admin client-app create … → returns clientAppId, API key and webhook secret, shown once (only hashes stored)
2. Give it numbers Assign AT numbers from the global pool to the client app's pool voice-admin numbers allocate <e164> --client <id>
3. Client stores credentials The client keeps the API key + webhook secret in its own secrets/env Client side
4. Verify Client calls GET /v1/ping (auth check); engine sends a signed test.ping webhook to check the client's receiver voice-admin client-app test-webhook <id>
5. Consume Client creates its accounts (POST /v1/accounts {externalRef}), assigns numbers from its pool to accounts, saves flows, adds agents, runs campaigns… /v1/** with Authorization: Bearer <key>
6. Rotate key Issue a second key; both work; client switches; old key revoked voice-admin key rotate <id> → voice-admin key revoke <id> <keyId>
7. Suspend (stop, reversible) API returns 403 CLIENT_SUSPENDED; running campaigns paused; no new outbound calls; webhooks held in the outbox (delivered after resume). Inbound calls keep working by default, so callers aren't hurt. Option hard: true also answers inbound with a system "service unavailable" message voice-admin client-app suspend <id> --reason … [--hard] · … resume <id>
8. Remove (permanent) Suspend → cancel campaigns → release numbers back to the global pool (AT callback config untouched; numbers reusable) → revoke keys → stop webhooks → optional export (calls, recordings, flows as a zip) → data deleted after a grace period (default 30 days) voice-admin client-app export <id> · … remove <id>

Isolation rules

Audit: every admin command and key use (first/last seen, IP) is logged in audit_log; voice-admin audit reads it.

9. Communication

Need Mechanism
Textify ⇄ engine commands/queries REST
Engine → Textify facts Webhooks via Postgres outbox + @Scheduled dispatcher
Engine → customer systems webhook node: async via endpoint_deliveries (retries), sync direct (Ph 2)
AT call control Callbacks + XML; AT REST call API
Ring agents LiveKit data messages (lobby rooms)
Presence, claims, waiting orders, concurrency, cache invalidation Redis
Background work Postgres jobs + workers (FOR UPDATE SKIP LOCKED)
Broadcast queue campaign_recipients table + dispatcher

Not used: MQTT, Kafka, RabbitMQ. Add Redis Streams first only if several engine instances + high event volume demand it.

10. APIs

10.1 Client API (Textify → engine) · Authorization: Bearer <api key> · base https://voice-api.textify.africa/v1 [VERIFY domain]

Area Endpoints
Accounts POST /accounts · PATCH /accounts/{id}
Numbers GET /numbers?status=UNASSIGNED · POST /numbers/{id}/assign · POST /numbers/{id}/release · PUT /numbers/{id}/hours · PUT /numbers/{id}/flow (validate + new version) · GET /numbers/{id}/flow/versions · POST /numbers/{id}/flow/rollback {version} · POST /numbers/{id}/test-call {toE164}
Flow tools POST /flows/validate (dry run, returns errors + premium char estimate)
Media POST /accounts/{id}/media (multipart) · POST /accounts/{id}/media/generate {text, lang, voice, tier} (preview) · GET /accounts/{id}/media · GET /media/{id} · DELETE /media/{id} · GET /voices
Agents PUT /accounts/{id}/agents/{ref} · DELETE … · PUT …/status · POST …/lobby-token
Departments POST/PUT/DELETE /accounts/{id}/departments[/{did}]
Calls POST /calls/outbound · POST /calls/{id}/accept · `POST /calls/{id}/hold
Campaigns POST /campaigns · POST /campaigns/{id}/recipients (chunks ≤ 10k) · POST /campaigns/{id}/estimate · `POST /campaigns/{id}/start
Opt-outs GET/POST/DELETE /accounts/{id}/opt-outs
Endpoints POST/PUT/DELETE /accounts/{id}/endpoints[/{eid}] · POST /endpoints/{eid}/test
Knowledge (Ph 3) PUT /accounts/{id}/knowledge · POST /accounts/{id}/knowledge/documents · GET /accounts/{id}/knowledge/unanswered
Stats GET /numbers/{id}/stats (per node) · GET /campaigns/{id}/stats
Health GET /ping (auth check)
Ops: CLI voice-admin on the box (primary). Optional /ops HTTP API bound to localhost/VPN only, ops token + admin IPs POST /ops/client-apps · POST /ops/client-apps/{id}/keys · DELETE /ops/client-apps/{id}/keys/{keyId} · POST /ops/client-apps/{id}/test-webhook · `POST /ops/client-apps/{id}/suspend

10.2 Engine → Textify

Event Data
call.started · call.answered callId, accountRef, direction, from, to, number, handledBy, agentRef, department
call.ended status, usage {pstnSeconds, trunkSeconds, waitSeconds, aiSeconds, atAmount}, recorded, endReason, flowVersion
call.missed callId, from, to, numberLabel, department
call.action_triggered callId, node, endpoint, status
message.requested callId/campaignId, to, channel, template, params → Textify sends SMS/WhatsApp
voicemail.received callId, from, mediaId/recording
recording.ready · summary.ready callId (+ summary JSON)
media.ready · media.failed mediaId, duration / error
tts.usage accountRef, chars, tier, ref (flow version / campaign)
campaign.progress · campaign.recipient_results · campaign.paused · campaign.completed counts / batches / reason / totals
kb.unanswered (Ph 3) accountRef, questions[]
test.ping sent on request to verify the client's webhook receiver

10.3 Hooks into the engine

Endpoint From
POST /hooks/at/{secret}/voice · /step/{callId}/{node} · /wait/{callId} AT
POST /hooks/livekit LiveKit
POST /hooks/openai (Ph 3) OpenAI
GET /media/{id}.mp3?t=… · GET /media/tts/{hash}.mp3?t=… AT / LiveKit Ingress

11. Infrastructure

11.1 Dedicated Voice box (Docker Compose; host networking for media)

Service Notes
caddy / nginx TLS for voice-api.… (API, hooks, media URLs) and wss://voice-rtc.… (LiveKit)
livekit livekit/livekit-server, 7880 on localhost
sip livekit/sip, 5060 + RTP 10000–20000, AT only
ingress livekit/ingress, agent-hold audio [VERIFY URL input]
redis shared by LiveKit + engine (separate DB index)
postgres engine DB (pgvector extension later for KB)
minio private buckets voice-media, voice-recordings; console on localhost
tts-basic self-hosted light TTS model (CPU), HTTP on localhost (if Basic passes the test)
voice-engine Spring Boot jar + ffmpeg
(fallback) egress only if AT recording isn't sufficient

Start 4 vCPU / 8 GB, measure. Region with lowest RTT to AT SIP servers.

11.2 Ports

Port Open to Purpose
22 owner IPs only (key-only login) SSH, voice-admin CLI
80/443 world API, hooks, media, WSS
7881/tcp, 50000–60000/udp world WebRTC (browser agents)
5060 udp/tcp, 10000–20000/udp AT IPs only SIP + RTP
7880, 6379, 5432, 9000/9001, tts-basic, 8080 localhost internal
(Ph 3) out 5061 + RTP to OpenAI CIDRs OpenAI AI over SIP

11.3 Env

AT_USERNAME / AT_API_KEY / AT_CALLBACK_SECRET / AT_IPS
LIVEKIT_URL / LIVEKIT_PUBLIC_WS_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET / LIVEKIT_SIP_HOST
DB_URL / DB_USER / DB_PASSWORD · REDIS_URL / REDIS_PASSWORD
MINIO_ENDPOINT / MINIO_ACCESS_KEY / MINIO_SECRET_KEY
MEDIA_URL_SECRET
TTS_BASIC_URL · TTS_PREMIUM_PROVIDER (openai|azure) / keys
OPENAI_API_KEY (Ph 2) · OPENAI_WEBHOOK_SECRET / OPENAI_SIP_URI (Ph 3)

11.4 LiveKit bootstrap (once, idempotent)

  1. Inbound trunk at: allowed_addresses = AT IPs, no number filter.
  2. Dispatch rule dispatch_rule_callee, room_prefix: "call-" → sip:{callId}@… ⇒ room call-{callId} [VERIFY].
  3. (Ph 3) Outbound trunk openai (TLS + SRTP).

12. Observability & testing

13. Tech stack & tools

Layer Tool Used for
Engine Java 21, Spring Boot 3.x (Web, Validation, JDBC/JPA, Scheduling, Actuator) All engine logic
DB PostgreSQL 16 + Flyway (pgvector later) State, flows, outbox, jobs, campaign queue, KB
Cache/locks Redis 7 Presence, claims, waiting orders, concurrency
Telephony Africa's Talking Voice (callbacks + XML: Play, GetDigits, Record, Dial; call API); optional africastalking-java for REST PSTN both directions
Media LiveKit server, SIP, Ingress; io.livekit:livekit-server JVM SDK; LiveKit JS SDK in Textify UI Rooms, browser agents
Voice (TTS) Basic: self-hosted light model (e.g. Piper, CPU) [VERIFY licence/quality]; Premium: OpenAI TTS or Azure Speech Text → audio files
AI OpenAI REST (transcribe, summarize); Ph 3 Realtime over SIP + java.net.http.WebSocket Summaries, live AI
Storage MinIO via AWS SDK v2 Media library, generated voice, recordings
Audio processing ffmpeg Transcode, normalize, trim
HTTP Spring RestClient AT, Textify, endpoints, TTS, OpenAI
Security Hashed API keys, HMAC-SHA256, LiveKit WebhookReceiver, SSRF guard, ufw, CrowdSec Service auth
Edge Caddy (or nginx) TLS, reverse proxy
Deploy Docker Compose, dedicated VPS Everything
Observability Logback JSON, Actuator + Micrometer (Prometheus) Logs, metrics
Testing JUnit 5, Testcontainers, WireMock, browser test page Unit + integration
Not used AT Say, MQTT, Kafka, RabbitMQ, OAuth/Keycloak, Python, browser extension Not needed

14. Repository layout

textify-voice/
├── docs/          SPEC.md, FLOWS.md, ARCHITECTURE.md
├── voice-engine/  Spring Boot (Java 21)
├── tts-basic/     self-hosted TTS sidecar (if Basic passes)
├── deploy/        docker-compose.yaml, livekit.yaml, sip, ingress, Caddyfile, firewall.sh, .env.example
├── audio/         system prompts (sw/en): notices, hold default, sorry, invalid option, numbers
└── agent-test/    minimal HTML page with LiveKit JS for the PoC

15. Open items to verify

  1. AT: Dial to sip:{callId}@host; IPs; codecs/DTMF; billing; Dial-to-SIP recording; CPS/concurrency; per-second vs per-minute; multi-number call API; callback signature (SPEC §11).
  2. AT: GetDigits wrapping Play (key interrupt, timeouts, callbacks per call) for the wait loop; Play fetch timeout/streaming; XML continuation after Dial.
  3. LiveKit: callee dispatch with arbitrary user part; trunk without number filter; Ingress URL input; JVM SDK method names.
  4. TTS: Basic candidate quality + commercial licence; Premium choice; Swahili normalizer coverage.
  5. Ph 3: LiveKit → OpenAI SIP interop; Realtime vs Live API; Swahili AI quality + cost.

Appendix A: Phase 3 AI (Java-only)

Ringmaster Voice: Product & Functional Specification

Status: draft for implementation · Owner: Joshua Saimon Sakweli (QBIT SPARK) · Updated 2026-11-08 Companion docs: FLOWS.md (flow & action reference) · ARCHITECTURE.md (components, data model, APIs, infra). Written so a new engineer, or a new AI coding session, can pick up the work with no prior context.

1. Background

Textify Africa (textify.africa) is an existing bulk SMS + WhatsApp platform, written in Go, live in Tanzania, Uganda and Nigeria. Its main model is white-label: partners brand it as their own and onboard their own customers, who top up a prepaid wallet.

We're adding Voice as a new Textify module, so the same Textify customers use it from the same portal and wallet. It combines:

Voice runs on a separate service, the Voice Engine (Java Spring Boot). It is an engine: blind to users, logins, roles and money. It is told what to do with numbers and calls and reports what happened. Textify consumes it like any other API. Other QBIT SPARK products (JikoXpress, Mauzodukani, NexGate) can consume it later.

Status: we already have a working Tanzanian number from Africa's Talking.

2. Goals

  1. Customers get a number, choose how it answers (Direct, Menu, later AI), and calls reach agents answering in the browser.
  2. Callers waiting for an agent hear hold music, never silence; nobody waits forever (voicemail).
  3. Click-to-call outbound for agents.
  4. Broadcast campaigns: voice message to thousands, keypad actions (talk to us, confirm via webhook, opt out…), retries, SMS fallback for non-answerers.
  5. Customers create what callers hear by uploading/recording audio or typing text (Basic or Premium voice).
  6. Keypresses can trigger the customer's own systems (webhooks with parameters).
  7. Every call is logged, optionally recorded, and (Phase 2) transcribed + summarized.
  8. Missed calls / voicemails become follow-ups in Textify (tickets, SMS/WhatsApp).
  9. Predictable costs: per-minute call billing from the Textify wallet; voice-generation cost known before it's spent.
  10. (Phase 3) AI receptionist with a per-business knowledge base, handing over to humans.

3. Non-goals

4. Key terms

Term Meaning
PSTN Normal phone network (Vodacom, Airtel, Yas, Halotel, TTCL)
AT Africa's Talking: our numbers, PSTN both directions, voice callbacks/XML
Voice callback / XML AT POSTs to our URL; we reply with XML actions (Play, GetDigits, Dial, Record…)
SIP / SIP trunk Internet call protocol / the AT → LiveKit connection carrying calls
LiveKit room Live audio session; each connected call = one room (caller + agent, later AI)
Browser agent Staff answering in the Textify web app
Department A group of agents that calls ring (a ring group)
Flow Graph of nodes deciding what happens on a call (FLOWS.md)
Node / action One step: play, menu, collect, connect, voicemail, webhook, branch, sms, optout, hangup, ai
Message Something callers hear: uploaded audio, or text turned into audio (Basic/Premium voice)
Campaign A broadcast: a flow run for every answered call on a recipient list
Endpoint A customer's saved webhook URL + secret, used by webhook actions
Knowledge base (KB) Business facts the AI uses (Phase 3)
DTMF Keypad presses

5. Product structure (what customers see in Textify)

Voice
├── Numbers          get/assign numbers
├── Call Center      inbound + agents
│   ├── Answer mode: Direct | Menu | AI (Ph 3)
│   ├── Departments & agents
│   ├── Business hours, closed message
│   ├── Waiting: hold music, comfort message, max wait → voicemail
│   ├── Recording on/off
│   └── Click-to-call (agents calling one customer)
├── Campaigns        bulk outbound
│   ├── Message (audio or text, variables)
│   ├── Keypad actions (talk to dept, webhook, sms, opt-out…)
│   ├── Recipients picked from Textify's EXISTING contacts / groups (same as SMS)
│   ├── Schedule, calling hours, retries
│   └── Results + SMS fallback to non-answerers
├── Audio library    uploads, recordings, generated voices
├── Integrations     webhook endpoints
├── Knowledge base   (Ph 3, for AI)
└── Calls & Tickets  call log, recordings, summaries; follow-up tickets (Textify-only feature)

Answer-mode templates generate flows; customers never see raw nodes (an advanced flow editor may come later; the format already supports it).

Answer mode Caller experience AT actions
Direct Greeting → ring department → wait with music → voicemail Play → (wait loop) → Dial → Record
Menu Greeting → "press 1 sales, 2 support" → that department → voicemail Play + GetDigits → wait loop → Dial → Record
AI (Ph 3) AI answers from the knowledge base, hands over to a department Dial → LiveKit + OpenAI → agents

6. Providers

Provider Role
Africa's Talking Numbers, PSTN in/out, callbacks + XML, outbound call API, call recording, SIP to LiveKit
LiveKit (self-hosted, dedicated box) Rooms, browser agents (WebRTC), agent-hold audio, Ph 3 AI participant. Separate from NexGate's LiveKit
TTS: Basic Light open-source model self-hosted on CPU (candidate: Piper Swahili voice), only if quality and commercial licence pass the listening test
TTS: Premium OpenAI TTS or Azure Speech (Swahili neural voices), chosen by listening test
OpenAI Ph 2 transcription + summaries; Ph 3 live AI (Realtime over SIP, controlled from Java)
MinIO (or Linode Object Storage) Audio library + recordings

7. Costs (AT Tanzania, africastalking.com/pricing, Oct 2026)

Item Price
Regular number (Airtel) Setup TZS 100,000 + 18% VAT; TZS 35,000/month + 18% VAT
Toll-free Same + TZS 170,000 deposit
Incoming / Outgoing TZS 17 / TZS 60 per min
Toll-free incoming TZS 200 / min
SIP trunk TZS 8.61 / min
Call type Approx. TZS / min
Inbound, menu / waiting / voicemail (stays on AT) 17
Inbound talking to browser agent 17 + 8.61 ≈ 25.61
Click-to-call 60 + 8.61 ≈ 68.61
Broadcast message only 60 per answered minute (unanswered likely free, confirm)
Broadcast → talk to agent 60 + 8.61 while talking

Voice generation (TTS): Basic = fixed server cost; Premium ≈ $15–16 per 1M characters (check current), always calculated before spending (at save or campaign creation). No AT <Say> usage. Waiting happens on AT (no trunk cost while waiting).

Agreed: cost follows minutes, not agents (unlimited agents); no mobile forwarding by default (77/min). Indicative selling prices (finalize after PoC): number 60–75k/month; inbound answered 40–50/min; outbound 90–100/min; broadcast 100–150 per answered call; premium voice per message/character; AI minute 150–250. Prices live in Textify.

8. Functional requirements

8.1 Numbers & answer modes

8.2 Flows & actions (full reference: FLOWS.md)

8.3 Messages & voice (what callers hear)

8.4 Inbound, waiting and agents

8.5 Outbound: click-to-call

8.6 Campaigns (broadcast)

8.7 Webhook actions (customer integrations)

8.8 Recording, transcript, summary

8.9 AI receptionist + knowledge base (Phase 3)

8.10 Follow-ups & tickets (Textify-only)

8.11 Usage & billing

9. Non-functional

Area Requirement
Availability Calls continue if Textify is down (fail-open balance check, queued events)
Isolation Dedicated Voice box (not shared with NexGate)
Latency AT callback response < 500 ms; agent ring < 3 s; connect after accept within one wait segment (~10–15 s max)
Security SIP only from AT; API keys + HMAC; verified webhooks; SSRF protection (ARCHITECTURE §8)
Privacy Recording notice, retention, own storage, signed URLs
Predictable cost No runtime TTS; premium cost known in advance
Simplicity REST + webhooks + Postgres + Redis + LiveKit + MinIO; no broker, no OAuth, no approvals

10. Phases

Phase 0: Proof of concept (on the working AT number)

  1. Callback → Play a file works.
  2. GetDigits menu works; keys arrive.
  3. Wait loop: GetDigits wrapping Play hold segment, callback every segment.
  4. Dial sip:{callId}@<livekit-sip-host> → room call-{callId} → browser test page hears the caller.
  5. Final callback fields (duration, amount, recordingUrl).
  6. Outbound call API → answer → Dial into LiveKit.
  7. Voice listening test: same Swahili sentences in Basic candidate (Piper), Premium candidates (OpenAI, Azure), and AT Say for reference, played on real phones to 5–10 Tanzanians.

Phase 1: Call center MVP Numbers; answer modes Direct + Menu; flow engine with nodes play, menu, collect, connect, voicemail, webhook (async), branch, sms, optout, hangup; messages (audio upload/record + text Basic tier); waiting v1; browser agents, departments, hold/transfer; click-to-call; recording to our storage; missed/voicemail events; usage events; balance check; test call; endpoints.

Phase 2: Bulk + intelligence Campaigns (all FR-C); Premium voice; sync webhooks; transcripts + summaries; stats per node/campaign.

Phase 3: AI ai node / AI answer mode; knowledge base; AI tools via endpoints; handover; improvement loop; AI outbound campaigns.

Later (already designed): position announcements, estimated wait, callback request, whisper, wrap-up time, advanced flow editor, partner white-label voice, other countries, other QBIT products as clients.

11. Questions for Africa's Talking (before Phase 1)

  1. Can XML Dial target an external SIP URI with a custom user part (sip:{callId}@our-host)? Transport, port, codecs (G.711), DTMF method?
  2. AT SIP signalling + media IPs (firewall, LiveKit allowlist).
  3. Billing for Dial-to-SIP: 17 (or 60 outbound) + 8.61? Anything else?
  4. Does Dial record="true" record Dial-to-SIP legs? Format, how long AT keeps the file?
  5. CPS and max concurrent calls (in + out) for our account; can they be raised?
  6. Outbound billed per second or per started minute? Unanswered/busy free?
  7. Outbound call API: multiple numbers per request? Returned session ids?
  8. Callback source IPs; any signature on callbacks?
  9. GetDigits wrapping Play: keys interrupt playback? Max timeout? Any limit on how many callbacks per call (our wait loop)?
  10. Play: timeout for fetching the URL; does it stream while downloading; any caching/pre-upload of media?
  11. After a Dial ends unanswered, does AT continue to the next XML (callback after Dial)?
  12. Airtel number: any issues for callers from Vodacom, Yas, Halotel, TTCL?
  13. TCRA: licence for reselling voice; rules for automated/marketing calls (consent, hours, caller ID). In writing.

12. Risks

Risk Mitigation
AT can't Dial to our SIP URI PoC step 4 first; fallback AT SIP softphones for agents
Wait loop limits (callbacks/timeouts) PoC step 3; fallback: wait inside LiveKit with ingress music (no keys while waiting)
Basic TTS not good enough / licence not commercial Listening test; Basic = Premium at lower price, or audio upload only
AT throughput slows broadcasts Confirm Q5; throttle; ETA
Swahili AI quality (Ph 3) Real-caller test; human handover always available
SIP attacks / webhook SSRF AT-only firewall; endpoint IP blocking
Textify outage Fail-open, outbox retries
Regulation (TCRA, PDPA) Written answers; notices; opt-out; retention

13. Decisions log

# Decision
D1 Separate Voice Engine (Java 21, Spring Boot), blind to users/auth/money; Textify (Go) consumes it via REST + webhooks
D2 AT is the only PSTN path, both directions
D3 Greeting/menu/collect/voicemail/waiting run on AT XML; anything live goes Dial → LiveKit
D4 All agents answer in the browser via LiveKit
D5 Adapters: Telephony (AT), Media (LiveKit), TTS (Basic/Premium), AI (OpenAI): plain Java interfaces
D6 No Python; Ph 3 AI = OpenAI Realtime over SIP controlled from Java (Python worker only as fallback)
D7 Service auth: API keys in, HMAC webhooks out, secret URL + IP allowlist for AT, built-in signatures for LiveKit/OpenAI
D8 Communication: REST + webhooks + Postgres outbox/jobs + Redis + LiveKit data messages; no broker in v1
D9 Engine owns media (MinIO, ffmpeg, signed play URLs); Textify references mediaId
D10 Messages = audio or text; text → audio file at save / before dialing; no AT <Say>, no runtime TTS
D11 Voice tiers Basic (self-hosted light model) / Premium (OpenAI or Azure), decided by listening test
D12 One flow/action system (11 node types) for inbound, campaigns and AI (FLOWS.md)
D13 Customers pick answer-mode templates (Direct / Menu / AI), not a flowchart editor (later maybe)
D14 Waiting happens on AT (hold loop), not in LiveKit; LiveKit audio only for agent hold
D15 Webhook actions via saved endpoints; async in Ph 1, sync in Ph 2; called by the engine directly
D16 Knowledge base lives in the engine per account; prompt-stuffing first, search tool when large
D17 Tickets/contacts/screen-pop live in Textify only
D18 Dedicated Voice box, separate from NexGate's LiveKit
D19 No approval workflows
D20 Consumers are client apps with a managed lifecycle: registered only by the engine owner via the voice-admin CLI over SSH (no public ops API) → numbers allocated → API key + webhook secret shown once, stored hashed, handed over directly into the service's env → consume → rotate → suspend (inbound keeps working by default) → remove (export, release numbers, delete after grace) (ARCHITECTURE §8.1)
D22 Contacts are Textify's existing contact module, reused as-is (same lists/groups as SMS/WhatsApp). Call center adds no contact store. The engine only receives phone numbers + variable values per campaign and caller numbers on calls; Textify does name lookup and screen pop
D21 Browser agents need no extension or app install: a normal web page with WebRTC; optional PWA later

Ringmaster Voice Engine: Flows & Actions Reference

The single action system used by inbound numbers, campaigns and (Phase 3) AI. Companion to SPEC.md and ARCHITECTURE.md. [VERIFY] = confirm with Africa's Talking (AT) in the PoC.

1. Concepts

A flow is a small graph of nodes. Each node does one thing and has named outcomes that point to the next node (or end).

Runs in Starts when Flows
Inbound number A call arrives open flow + closed flow (chosen by business hours)
Campaign The recipient answers One campaign flow
AI (Ph 3) AI hands over / calls a tool Uses the same connect, webhook, sms actions

Customers normally don't edit nodes. Textify templates generate flows:

An advanced flow editor may come later; this format already supports it.

1.1 How AT actions map

AT action Used for
Play Every message callers hear (uploaded audio, generated text, hold music)
GetDigits Menus, collecting digits, keys during waiting
Record Voicemail
Dial Anything live: into LiveKit (sip:{callId}@…) for agents (and AI in Ph 3)
Say Not used (predictable cost, consistent voice)

Rule: one-way listening = Play, keys = GetDigits, live two-way = Dial → LiveKit.

2. Message (what callers hear)

One shape everywhere:

{ "source": "audio", "media": "med_123" }
{ "source": "text",  "text": "Karibu {name}", "lang": "sw", "voice": "female", "tier": "basic" }

3. Variables

Written as {name} inside message text and webhook params.

Group Variables Available
System {caller}, {number}, {callId}, {date}, {time}, {key} (last key), {businessName} Everywhere
Contact Any recipient column: {name}, {amount}, {appointment_id}… Campaign flows only
Collected {input.<saveAs>} from collect After that node
Webhook result {result.<field>} from a sync webhook After that node (Ph 2)

Missing variable at runtime → empty string + warning in the call trail (validation prevents most cases).

4. Node types (complete list)

Common fields: type, on (outcome → node id or "end"), optional label (for stats).

4.1 play

Play a message.

{ "type": "play", "message": {…}, "on": { "next": "menu" } }

Outcomes: next. XML: Play.

4.2 menu

Message + wait for one key.

{ "type": "menu", "message": {…},
  "timeoutSec": 8, "retries": 2,
  "invalidMessage": { "source": "audio", "media": "sys_invalid_option" },
  "on": { "1": "sales", "2": "support", "0": "menu", "noInput": "support", "invalid": "support" } }

Outcomes: keys 0–9, *, #, noInput, invalid (after retries). XML: GetDigits numDigits=1 wrapping Play.

4.3 collect

Ask for several digits and save them.

{ "type": "collect", "message": {…}, "saveAs": "account",
  "minDigits": 6, "maxDigits": 10, "finishOnKey": "#", "pattern": "^[0-9]{6,10}$",
  "timeoutSec": 10, "retries": 2,
  "on": { "ok": "lookup", "failed": "agent" } }

Outcomes: ok, failed. XML: GetDigits. Saved as {input.account}.

4.4 connect

Put the caller through live.

{ "type": "connect",
  "target": { "department": "dep_support" },        // or {"agent": "agentRef"} | {"external": "+2557…"} | {"ai": {…}} (Ph 3)
  "ringTimeoutSec": 30,
  "wait": {
    "music": { "source": "audio", "media": "sys_hold_default" },
    "comfortEverySec": 45,
    "comfortMessage": { "source": "text", "text": "Tafadhali endelea kusubiri, mhudumu atakupokea punde.", "tier": "basic" },
    "maxWaitSec": 300,
    "keys": { "1": "voicemail" },
    "announcePosition": false,     // later
    "announceWaitTime": false,     // later
    "callbackKey": null            // later: e.g. "2" = request a callback
  },
  "whisper": null,                  // later: message only the agent hears
  "record": true,
  "on": { "noAnswer": "vm", "voicemail": "vm", "callback": "cb_ok" } }

(The comfort message reads: "Please keep waiting, a staff member will answer you shortly.")

Outcomes: noAnswer (max wait reached / nobody available), plus any outcome named in wait.keys. Answered = the call is live; the flow ends there.

How it runs: see §6 "Waiting loop". external (forward to a mobile) is allowed but off by default (extra outbound leg cost).

4.5 voicemail

{ "type": "voicemail", "message": {…}, "maxSec": 60, "beep": true, "on": { "next": "end" } }

XML: Play + Record. Emits voicemail.received.

4.6 webhook

Call the customer's saved endpoint.

{ "type": "webhook", "endpoint": "ep_clinic", "mode": "async",
  "params": { "event": "appointment_confirmed", "id": "{appointment_id}", "phone": "{caller}", "key": "{key}" },
  "on": { "next": "thanks" } }

4.7 branch

Decide by condition (engine-side, instant).

{ "type": "branch",
  "cases": [
    { "if": { "hours": "closed" }, "go": "closedMsg" },
    { "if": { "var": "result.status", "equals": "paid" }, "go": "paidMsg" },
    { "if": { "var": "input.account", "matches": "^99" }, "go": "vip" }
  ],
  "on": { "default": "menu" } }

Operators: equals, in, matches (regex), exists, hours: open|closed. First match wins.

4.8 sms

Ask Textify to send an SMS/WhatsApp to the caller ("we've sent you the payment link").

{ "type": "sms", "channel": "sms", "template": "payment_link", "params": { "amount": "{amount}" },
  "on": { "next": "thanks" } }

Engine emits message.requested → Textify sends and bills (it owns messaging and templates).

4.9 optout

Add {caller} to the account's do-not-call list. Outcome next.

4.10 hangup

Optional goodbye message, then end.

{ "type": "hangup", "message": { "source": "text", "text": "Asante, kwaheri." } }

(The goodbye reads: "Thank you, goodbye.")

4.11 ai (Phase 3)

{ "type": "ai",
  "instructions": "Jibu maswali kuhusu bei na saa za kazi.",
  "knowledgeBase": true,
  "tools": { "endpoints": ["ep_orders"], "sms": ["payment_link"] },
  "handoff": { "department": "dep_support" },
  "maxMinutes": 10,
  "on": { "handoff": "agent", "ended": "end" } }

(The instructions read: "Answer questions about prices and working hours.")

Runs via Dial → LiveKit → OpenAI Realtime (ARCHITECTURE Appendix A). The AI uses the account knowledge base, can call endpoints/sms as tools, and hands over to a department (handoff outcome → usually a connect).

5. Examples

5.1 Menu answer mode (generated by the template)

{
  "version": 7,
  "settings": { "record": true, "recordingNotice": true },
  "open": {
    "start": "greet",
    "nodes": {
      "greet": { "type": "play", "message": { "source": "audio", "media": "med_karibu" }, "on": { "next": "menu" } },
      "menu":  { "type": "menu", "message": { "source": "text", "text": "Bonyeza 1 mauzo, 2 huduma kwa wateja.", "tier": "basic" },
                 "on": { "1": "sales", "2": "support", "noInput": "support", "invalid": "support" } },
      "sales":   { "type": "connect", "target": { "department": "dep_sales" },   "wait": { "music": { "source": "audio", "media": "sys_hold_default" }, "maxWaitSec": 180 }, "on": { "noAnswer": "vm" } },
      "support": { "type": "connect", "target": { "department": "dep_support" }, "wait": { "music": { "source": "audio", "media": "sys_hold_default" }, "maxWaitSec": 300, "keys": { "1": "voicemail" } }, "on": { "noAnswer": "vm", "voicemail": "vm" } },
      "vm": { "type": "voicemail", "message": { "source": "text", "text": "Acha ujumbe baada ya mlio.", "tier": "basic" } }
    }
  },
  "closed": {
    "start": "closedMsg",
    "nodes": {
      "closedMsg": { "type": "play", "message": { "source": "text", "text": "Tumefunga kwa sasa. Tunafungua saa mbili asubuhi.", "tier": "basic" }, "on": { "next": "vm" } },
      "vm": { "type": "voicemail", "message": { "source": "text", "text": "Acha ujumbe baada ya mlio.", "tier": "basic" } }
    }
  }
}

(The Swahili messages read: "Press 1 for sales, 2 for customer service." · "Leave a message after the tone." · "We're closed right now. We open at eight in the morning." Tanzanian time "saa mbili asubuhi" is 08:00.)

5.2 Campaign: appointment reminder

{
  "start": "msg",
  "nodes": {
    "msg": { "type": "menu",
             "message": { "source": "text", "text": "Habari {name}, una miadi kesho saa {time_slot}. Bonyeza 1 kuthibitisha, 2 kuongea na mhudumu, 3 kupata ujumbe wa SMS, 9 kujiondoa.", "tier": "premium" },
             "on": { "1": "confirm", "2": "agent", "3": "sms", "9": "out", "noInput": "msg", "invalid": "msg" } },
    "confirm": { "type": "webhook", "endpoint": "ep_clinic", "mode": "async",
                 "params": { "event": "confirmed", "id": "{appointment_id}", "phone": "{caller}" }, "on": { "next": "thanks" } },
    "thanks": { "type": "hangup", "message": { "source": "text", "text": "Asante, tumethibitisha miadi yako.", "tier": "basic" } },
    "agent":  { "type": "connect", "target": { "department": "dep_reception" }, "wait": { "music": { "source": "audio", "media": "sys_hold_default" }, "maxWaitSec": 120 }, "on": { "noAnswer": "vm" } },
    "vm":     { "type": "voicemail", "message": { "source": "audio", "media": "med_vm" } },
    "sms":    { "type": "sms", "channel": "sms", "template": "appointment_details", "params": { "id": "{appointment_id}" }, "on": { "next": "smsOk" } },
    "smsOk":  { "type": "hangup", "message": { "source": "text", "text": "Tumekutumia ujumbe. Kwaheri.", "tier": "basic" } },
    "out":    { "type": "optout", "on": { "next": "bye" } },
    "bye":    { "type": "hangup", "message": { "source": "text", "text": "Umeondolewa kwenye orodha. Kwaheri.", "tier": "basic" } }
  }
}

(The Swahili messages read: "Hello {name}, you have an appointment tomorrow at {time_slot}. Press 1 to confirm, 2 to talk to a staff member, 3 to get an SMS, 9 to opt out." · "Thank you, we've confirmed your appointment." · "We've sent you a message. Goodbye." · "You've been removed from the list. Goodbye.")

6. Waiting loop (connect to a department)

Waiting happens on AT, not in LiveKit. Reasons: keys work while waiting, no trunk cost while waiting, announcements are just files.

connect node:
  1. engine marks call WAITING_AGENT, adds it to the department's waiting order (oldest first)
  2. rings available agents in their browsers (LiveKit lobby data message)
  3. returns XML:  <GetDigits numDigits="1" timeout="1" callbackUrl=".../wait/{callId}">
                      <Play url="{hold segment ~10–15 s}"/>
                   </GetDigits>
AT calls back after each segment (or on a key):
  - agent accepted?       → <Dial phoneNumbers="sip:{callId}@sip.voice…" record="true"/>   → live
  - key pressed?          → outcome from wait.keys (e.g. voicemail)
  - waited ≥ maxWaitSec?  → outcome noAnswer
  - else                  → next segment: music, or comfortMessage every comfortEverySec

In-call hold/transfer by the agent happens inside LiveKit (hold music via LiveKit Ingress).

7. Validation rules (on save; errors shown in Textify, no approvals)

  1. Every outcome points to an existing node or end; start exists.
  2. ≤ 50 nodes per flow; menus ≤ 12 options; ≤ 3 retries.
  3. Variables exist in scope (contact columns only in campaigns; {input.x} only after its collect; {result.x} only after a sync webhook).
  4. connect targets, endpoints, media, sms templates belong to the same account; media READY; text generation succeeded.
  5. Message text ≤ 500 characters.
  6. Inbound numbers must have both open and closed flows (template provides defaults).
  7. ai nodes only when Phase 3 is enabled for the account.

Runtime guards: max 25 node visits per call (loop guard → system goodbye + hangup); max call length per flow (default 30 min).

8. Versioning

9. Execution (engine)

10. Cost per node

Node Cost
play, menu, collect, voicemail, hangup Call minutes only (on AT)
Text messages Basic: free · Premium: characters × rate, known at save / campaign creation
connect (waiting) Call minutes only (still on AT)
connect (talking) Minutes + SIP trunk 8.61/min; external = extra outbound leg
webhook, branch, optout Free
sms Textify's SMS/WhatsApp price
ai AI per-minute price (Ph 3)

11. Phase availability

Feature Phase
play, menu, collect, connect (department/agent), voicemail, webhook async, branch, sms, optout, hangup 1
Waiting: music, comfort message, max wait, keys while waiting 1
Basic voice tier 1
Premium voice, webhook sync, campaigns 2
ai node, knowledge base, AI tools 3
Position / wait-time announcements, callback request, whisper, wrap-up, connect.external on by default, advanced flow editor Later