Skip to main content

Textify Voice: Architecture

Companion to SPEC.md (what/why) and FLOWS.md (flow & action reference). This file is how: components, adapters, call flowspaths, (inbound,media outbound,& broadcast),voice, knowledge base, data model, APIs, auth, communication, infra.infra, tools. Stack: Textify (Go) → Voice Engine (Java 21, Spring Boot) → Africa's Talking + LiveKit (self-hosted) + TTS (Basic/Premium) + OpenAI + MinIO. No Python, no message broker. [VERIFY] = check against provider docs or Africa's Talking before coding against it.

1. Overview

                          ┌──────────────────────────────────────┐
  Phones (any TZ network) │        Africa's Talking (AT)          │
        ◄──── PSTN ──────►│ numbers · inbound callbacks · XML (Play,      │
                          │ GetDigits, Record, Dial) · call API · │
                          │ outbound call API · recordings · SIP to LiveKit           │
                          └───────┬───────────────────▲──────────┘
               voice callbacks (POST)   │                   │ XML replies,replies · call API (REST)· +fetch audio URLs
               Dial to SIPsip:{callId}  │                   │
┌──────────────────────── Voice box (dedicated) ──────┼─┼─────────────────────────────┐
│                                 ▼                   │                              │
│  ┌──────────────┐ SIP  ┌──────────────────────────────────────────────┐            │
│  │ livekit-sip  │◄──────┤              VOICE ENGINE (Spring Boot)       │──► Postgres  │Postgres│
│  └──────┬───────┘ (AT) │ accounts · numbers · flows · media · agents · │──► Redis     │
│          │               │ calls · campaigns · recordings · events  │──► S3 store   │
│  ┌──────▼──────────┐    │ calls · campaigns · endpoints · knowledge ·   │──► MinIO   │
│  │ livekit-server │◄──►│ TelephonyAdapterrecordings MediaAdapter· AiAdapter│events                           │            │
│  │ + ingress      (hold)│    │ Telephony│Media│Tts(Basic,Premium)│Ai adapters │──► tts-basic│
│  └───────▲─────────┘    └──────▲───────────────┬───────────────┘              │
│   └──────▲──────────┘  │(CPU REST (API key)   │ webhooks (HMAC)              model)│
└──────────┼───────────────────────┼──────────────────┼────────────────────────────────────┘
          │ WebRTC              │ ▼REST (API key) │ webhooks (HMAC)        ┌──────────────┐
  ┌───────┴────────┐     ┌──────┴───────────────▼───────────┐        ┌─────────┐             │ OpenAI /     │
  │ Agent browser  │◄─app─►app►│ TEXTIFY (Go)                     │             │ OpenAIAzure (TTS,  │◄─ AiAdapter
  │ (Textify UI +  │     │ users · tenants · wallet · │prices│             │ (Phtranscribe,  2–3)│
  │ LiveKit JSJS)    SDK)│     │ pricing · UI · contacts · tickets · SMS/WhatsAppWA │             └─────────┘│ summary, AI) │
  └────────────────┘     └──────────────────────────────────┘             └──────────────┘
                                         customer systems ◄── webhook actions (signed) ── engine

ThreeRules rulesthat hold the design together:together

  1. AT is the only road to the phone network,network, both directions. Inbound arrives as AT callbacks; outbound starts with AT's call API.
  2. TalkingListening to= aPlay, personkeys (or= AI)GetDigits, happensvoicemail in= Record, live = Dial → LiveKit. The engine's XML says Dial sip:{callId}@<livekit-sip-host>Say andis not used.
  3. Waiting for an agent happens on AT (hold loop); the call enters room call-{callId}. LiveKit only everonce receivesan callsagent fromaccepted.
    4. AT.
  4. No TTS during live calls. Text becomes audio files at save time / before dialing.
  5. The engine is blind.blind It knows: accountId, agentRef, numbersnumbers, andcalls. calls, neverNever users, rolesroles, prices, tickets or money.contacts.

1.1 Responsibilities

Component Owns Never owns
Textify (Go) Users, login, Users/login/roles, tenants, partners, wallet, prices, all UI screens(answer-mode (flowtemplates, editor,campaign campaigns,screens, dashboard)audio library UI, KB editor), contactcontacts, lists,tickets, screen-pop data, SMS/WhatsApp follow-upssending Telephony / LiveKit / OpenAITTS / AI logic
Voice Engine Client apps, accounts (opaque), numbers, flows,flows + versions, media + generated voice, agents (opaque refs), ring groups,departments, presence, waiting orders, calls, campaigns, opt-outs, endpoints, knowledge base, recordings, transcripts, usage events Users,People, authmoney, of people, prices, moneytickets
AT PSTN, numbers, XML execution, outbound dialing, recording Anything else
LiveKit Rooms, WebRTC to browsers,WebRTC, SIP from AT, agent-hold audio, (Ph 3)3 AI participant Business logic
TTS providersTurning text into audio filesState
OpenAI Ph 2:2 transcribe + summarize.transcribe/summarize; Ph 3:3 live voiceAI State, routing

2. Inside the engine

Stack: Java 21, Spring Boot 3.x, PostgreSQL + Flyway, Redis, @Scheduled jobs,workers, virtual threads.threads, ffmpeg in the image.

2.1 Adapters (plain interfaces, one implementation each,interfaces; no plugin framework)

interface TelephonyAdapter {                // impl: AfricasTalkingAdapter
  String render(List<ActionXmlAction> actions);                    // → AT XML (Play, GetDigits, Record, Dial, Reject)
  List<OutboundResult> placeCall(placeCalls(String from, List<String> to);   // AT call API
  InputStream downloadRecording(String url);
}

interface MediaAdapter {                    // impl: LiveKitAdapter
  String sipUriFor(UUID callId);                             // sip:{callId}@sip.voice…
  String roomToken(String room, String identity, boolean canPublish)publish);
  void sendData(String room, Object message)msg);                    // ring / ring_cancel to/ lobbiesconnecting
  void startHoldAudio(String room)room, String url); void stopHoldAudio(String room);   // agent hold
  void removeParticipant(String room, String identity); void deleteRoom(String room);
  // Ph 3: dialAi(room, callId)
}

interface TtsAdapter {                      // impls: BasicTtsAdapter (self-hosted), removeParticipant(room,PremiumTtsAdapter identity)(OpenAI or Azure)
  byte[] synthesize(String text, String lang, String voice);
  List<VoiceOption> voices();
}

interface AiAdapter {                       // impl: OpenAiAdapter
  Transcript transcribe(InputStream audio, String langHint); // Ph 2
  Summary summarize(Transcript t)t, String accountContext);    // Ph 2
  // Ph 3: acceptLiveCall(openAiCallId, SessionConfig), control WebSocket, hangup
}

A TtsRouter picks the adapter by tier. Swapping latera (another telco, another AI/TTS)provider = a new class,class nothing+ else changes.config.

2.2 Packages

com.qbitspark.voice
├── api          // client REST /v1/** (API-key auth)
├── hooks        // /hooks/at/{secret}/**, /hooks/livekit, /hooks/openai
├── telephony    // AfricasTalkingAdapter,AT adapter, XML builder (Say, Play, GetDigits, Dial, Record, Reject)
├── media        // LiveKitAdapter,LiveKit adapter, tokens, lobby/ring, ingress hold(agent audiohold)
├── aivoice        // OpenAiAdapterTtsRouter, Basic/Premium adapters, Swahili text normalizer, generation cache
├── library      // media assets: upload, validate, ffmpeg transcode, signed play URLs
├── flows        // flow JSON model, validator, executorversions, NodeExecutor per node type (node → XML)FLOWS.md)
├── callswaiting      // callhold lifecycle,loop, statewaiting machine,order timelineper department, segment builder
├── routing      // ring groups,departments, presence, waitingring list,strategies, claim (first accept wins)
├── calls        // lifecycle, state machine, context, trail
├── campaigns    // broadcastcampaigns, campaigns,recipients, pre-generation, dispatcher, retries, opt-outs
├── endpoints    // customer webhook endpoints, signer, SSRF guard, sync/async calls
├── knowledge    // KB storage, prompt builder (Ph 3), search tool (later)
├── ai           // OpenAI adapter
├── recording    // download → S3,MinIO, retention
├── jobs         // Postgres job table + @Scheduled workers (transcribe, summarize, cleanup)
├── events       // outbox + signed webhookdispatcher dispatcherto client apps
├── accounts     // client apps, accounts, numbers, agents
└── common

2.3 Flow model (stored per number, built in Textify's UI)

{
  "record": true,
  "hours": { "tz": "Africa/Dar_es_Salaam", "open": "Mon-Fri 08:00-17:00, Sat 09:00-13:00",
             "closed": { "say": "Tumefunga kwa sasa. Acha ujumbe.", "next": "vm" } },
  "start": "greet",
  "nodes": {
    "greet": { "type": "play", "url": "https://…/karibu.mp3", "next": "menu" },
    "menu":  { "type": "menu", "say": "Bonyeza 1 mauzo, 2 huduma kwa wateja",
               "options": { "1": "sales", "2": "support" },
               "noInput": "support", "retries": 2 },
    "sales":   { "type": "ring", "group": "grp_sales",   "timeoutSec": 30, "noAnswer": "vm" },
    "support": { "type": "ring", "group": "grp_support", "timeoutSec": 60, "noAnswer": "vm" },
    "vm":    { "type": "voicemail", "say": "Acha ujumbe baada ya mlio", "maxSec": 60 }
  }
}

(The Swahili prompts read: "We're closed right now. Leave a message." · "Press 1 for sales, 2 for customer service" · "Leave a message after the tone".)

Node types: say, play, menu, ring, voicemail, hangup (Ph 3: ai). The engine validates on save (every next exists, no loops without input).

2.4 Executing a flow with AT callbacks

  • Each number's voice callback URL in AT points to https://voice-api…/hooks/at/{secret}/voice.
  • The executor finds the number → flow → current node, and returns XML for that node.
  • State travels in the callback URL of each action (e.g. GetDigits callbackUrl=/hooks/at/{secret}/step/{callId}/menu), plus the calls row. No session memory needed.
  • The final callback (isActive=0) carries duration, cost (amount, currencyCode), recordingUrl, hangup cause → close the call. [VERIFY] exact AT field names.
  • XML responses must be fast (< 500 ms): number/flow config cached in memory, refreshed on change.

3. Call flowspaths

3.1 Inbound → menu → waiting → browser agent

Caller  AT                  Engine                                    LiveKit           Agent browser
 │dial─►│── POST /voice ──────►│ number→account→flownumber → account → flow(open/closed)    │                    │
 │      │                   │ can-call? (Textify, 1.5s,5 s, fail-open)    │                    │
 │      │◄── XML: SayPlay notice + Play greetinggreeting, +GetDigits(menu) GetDigits ───────────│                    │
 │ "2" ►│── POST /step/{id}/menu (dtmfDigits=2) ────────────────────────────────────►│ node=supportconnect (ring)  │
  │      │◄── XML: Dial sip:{callId}@sip.voice… record=true ───│                      │
  │      │══ SIP INVITE (user part = callId) ════════════════════════► room call-{callId}
  │      │                      │◄─ webhook participant_joined ─│dep_support │
 │      │                   │ startadd holdto audiowaiting (ingress)order; ──►│                      │
  │      │                      │ pick availablering agents ───────────── ring data msg ─► lobby ─────►│ popup
 │      │◄── GetDigits(timeout 1){ Play hold segment } → /wait/{id} ──│                    │◄
 │ ♪♪   │── POST /wait/{id} (segment ended) ─────────────────────────►│ acceptnot accepted yet    │
 │      │◄── next segment (viamusic Textify)/ comfort message) ──────────────────│◄──── accept ───────│ (via Textify)
 │      │                   │ claim (Redis SET NX) → room token ───────────────────►│
  │      │                      │ stop hold audio ─────────────►│◄ joins room
 │      │── POST /wait/{id} ─►│ accepted → Dial sip:{callId}@… record │                    │ "Connecting…"
 │      │══ SIP INVITE ═════════════════════════════════════════════════► joins,room talkscall-{id} ◄══│
 │◄═════╪══════════════════════╪═ live audio ═════════ audio ══════════════════╪══════════════════════│
 │ hangup                      │                               │                      │
  │      │hangup│── POST /voice (isActive=0: duration, amount, recordingUrl) ►│ close call → eventsevents, jobs │
  • Waiting:Agents ifaccept no agent is available,while the caller stays on hold in the room; when an agent becomes AVAILABLE (or finishes a call), the engine rings again. Order = oldest waiting call first (Redis sorted set per group).
  • Timeout → noAnswer node. Since the call is nowmid-segment; inconnection LiveKit,happens voicemail there = the engine plays the prompt via ingress and records via egress, or simpler: remove the SIP participant so AT's Dial ends, then AT continues withat the next XMLcallback (segments 10–15 s).
  • RecordmaxWaitSec reached / key pressed → outcome (voicemail etc.) on theAT. Dial's callback. [VERIFY] AT continues the flow after Dial ends (prefer this: keeps voicemail on AT).
  • Missed: caller hangs up or nobody handled itUnanswered → call.missed.
  • Agent hold/transfer inside LiveKit: engine starts Ingress hold audio into the room.
  • [VERIFY] wait-loop behaviour (FLOWS §6); fallback = Dial immediately, hold music via Ingress.

3.2 Outbound click-to-call

Agent clicks number → Textify → POST /v1/calls/outbound {accountId, fromNumberId, to, agentRef}
Engine: 1.can-call can-call(OUTBOUND)
  2.→ create call + room call-{callId}; return→ agent token →(agent browserjoins, joins (hears ringback tone fromin page)
        3.→ TelephonyAdapter.placeCall(AT call API (from, to) → AT dials the customer; store AT sessionId → callId
Customer answers → AT callback (direction=Outbound, sessionId) → engine finds callId
  → XML:XML Dial sip:{callId}@sip.voice…@… → customer joins the agent's room → talk
End → final callback → call.ended
(pstnSeconds outbound, trunkSeconds)

3.3 Broadcast campaignCampaign

Textify → POST /v1/campaigns {accountId, fromNumberId, audioUrl|ttsText, keys:{1:"grp_sales",9:"optout"},
                              window, cps, maxConcurrent, retries} (+ recipients in chunks) → POST /start
Engine:
  1. validate flow (chunkspinned ofversion), ≤10k)opt-outs, variables vs recipient columns
  2. PRE-GENERATE voice: render each recipient's text, dedupe by hash, TtsRouter → files (jobs)
     cost estimate (premium chars) available before start; failure → campaign PAUSED (TTS_FAILED)
  3. Dispatcher (@Scheduled every1 1s, per campaign)s):
  - inside calling window? not paused? concurrentactive < max?
  -maxConcurrent?
       SELECT … FROM campaign_recipients WHERE status='PENDING' AND next_attempt_at<=now()
       ORDER BY id LIMIT :cps FOR UPDATE SKIP LOCKED -→ skipAT opt-outscall API → markDIALING
  OPTED_OUT4. - placeCall(from, batch) → mark DIALING, store AT sessionIds
Recipient answersAnswer → AT callback → XML:run Playcampaign audio + GetDigits(keys)
  - "1" → Dial sip:{callId}@livekit → ring groupflow (sameFLOWS as§5.2) 3.1)with -the "9"recipient's →variables
  add to opt_outs → Say "Umeondolewa" ("You've been removed") → hangup5. Final callback → recipient ANSWERED / NO_ANSWER / BUSY / FAILEDFAILED; (+seconds,retries amount)per - NO_ANSWER/BUSY & attempts < max → PENDING with next_attempt_at = now + retryDelaypolicy
Events: campaign.progress every(30 30s (counts)s) · campaign.recipient_results batches(batches) · campaign.paused · campaign.completed
Textify: SMS fallback to NO_ANSWERNO_ANSWER; numbers;pause pauseson campaign (POST /pause) iflow wallet
low

The campaign_recipients table is the queue. No broker needed.

3.4 RecordingAfter →the transcript → summarycall (Phase 2)jobs)

final callback recordingUrl → job DOWNLOAD → S3MinIO s3://rec/{accountId}/{callId}.mp3voice-recordings → recording.ready
  → job(Ph 2) TRANSCRIBE (OpenAI, sw/en) → job SUMMARIZE (OpenAI, JSON) → summary.ready
voicemail → voicemail.received (+ recording)
daily job RETENTION → delete whereexpired delete_after < now()recordings

Recording comes from AT (Dial record="true"), soRecord). noNo LiveKit egress service is needed. [VERIFY] AT records the Dial-to-SIP leg;SIP; fallback = LiveKit egress.

Summary JSON: {summary, intent, resolved, followUps[], sentiment, topic, language}.

3.5 AI (Phase 3),3 same patternAI

Flow node ai (or noAnswer: ai, or campaign key 1: ai)node → call is dialedDial into LiveKit as usual → engine dials OpenAI Realtime into the room over SIP (LiveKit outbound trunk, TLS 5061 + SRTP) → OpenAI webhook → engine accepts with instructions/instructions built from the knowledge base + node instructions + tools → controls via WebSocket from Java → transfer_to_humanhandoff rings agentsa →department (agent joins → engine hangs up theroom, AI leg.leg Detailshung keptup) in→ summary job. Appendix A.

3.6 Call state machine

INBOUND:  RINGING → IN_FLOW → WAITING_AGENT → CONNECTING → ANSWERED → COMPLETED
OUTBOUND: DIALING → IN_FLOW (campaign) / CONNECTING (click-to-call) → ANSWERED → COMPLETED
any → MISSED | NO_ANSWER | BUSY | FAILED | REJECTED_NO_BALANCE
handled_by: NONEFLOW_ONLY | FLOW_ONLYVOICEMAIL | AGENT | AI | AI_THEN_AGENT

4. Browser agents

  • AgentNo browser extension or app install needed. The agent UI livesis a normal page in Textify's web app, using the LiveKit JS SDK.SDK; browsers have WebRTC built in (Chrome, Edge, Firefox, Safari).
    • Requires: HTTPS, microphone permission, the Textify tab open (can be in the background).
    • Browsers block sound until the user interacts with the page, so agents click "Start shift / Go available" once; that click unlocks the ringtone.
    • Ring alerts when the tab is in the background: Web Notifications API + ringtone; optional PWA (installable web app) for a desktop-app feel. A Chrome extension or native/mobile app is not required; consider only later (e.g. ringing with the browser closed).
  • Lobby room per agent:agent lobby-{accountId}-{agentRef}, connected with (mic offoff) while logged in.available:
    • Presencepresence = lobby participant_joined/leftjoin/leave webhooks (automatic online/offline) + explicit status from Textify.status;
    • Ringingringing = engine sendData(lobby, {type:"ring", callId, from, numberLabel, groupName}department}).
  • Accept → browser → Textify backend → POST /v1/calls/{id}/accept {agentRef} → engine claims (SET call:{id}:agent NX EX 60)claim → returns call-room token → browserjoin joins(shows with"Connecting miccaller…" on.until the caller arrives). Losers: 409 + ring_cancel.
  • BrowserScreen neverpop: holdsthe browser uses from to ask Textify for contact + open tickets. The engine credentials;never onlysees short-livedthem.
  • Contacts: Textify's existing contact module (the one SMS/WhatsApp already use) is reused for the call center: campaign recipients, caller names, customer timeline. The engine stores no contacts; campaigns receive {e164, vars} only.
  • In-call: mute (client-side), hold (engine starts Ingress music, mutes agent), transfer (engine re-rings target; caller stays in room tokenswith (≈10hold minmusic), tohang join, room-scoped).up.

5. Media library & voice generation

5.1 Uploaded audio

Textify (file or browser recording) → POST /v1/accounts/{id}/media (multipart)
Engine: validate (≤ 10 MB, ≤ 5 min, magic bytes) → MinIO original/ → job TRANSCODE (ffmpeg:
        mono, 16 kHz, MP3, loudness-normalized, trimmed) → MinIO play/ → READY → media.ready

5.2 Text → voice (generated audio)

  • Messages with source: text are generated at flow save (inbound) or before dialing (campaigns, per-recipient text).
  • Pipeline: render variables → Swahili normalizer (numbers, currency, dates, times → words) → cache lookup hash(text+lang+voice+tier) → TtsRouter.tier → adapter → ffmpeg normalize → MinIO generated/ → media_assets row (kind GENERATED).
  • Basic: self-hosted light model in a small sidecar (tts-basic, CPU), e.g. Piper with a Swahili voice, only if quality + commercial licence pass the listening test. [VERIFY licence]
  • Premium: OpenAI TTS or Azure Speech (Swahili neural voices), picked by listening test.
  • Costs predictable: premium character count computed before generation → shown in Textify → reported (tts.usage).
  • No runtime fallback to AT Say. Failure → flow save error (previous version stays live) / campaign paused. Call-time safety net: system file.
  • On-demand generation exists only for sync webhook responses (Ph 2): Play url=/media/tts/{hash}.mp3 generated on first fetch, then cached. [VERIFY] AT Play fetch timeout.

5.3 Playing to AT

  • XML uses <Play url="https://voice-api…/media/{id}.mp3?t={hmac,24h}">; the engine streams from MinIO (cache headers). Buckets stay private.
  • Same URLs feed LiveKit Ingress (agent hold).
  • Deleting media used by an active flow / running campaign → 409. Per-account storage quota.

5.4 Buckets

voice-media/system/{lang}/{name}.mp3                 # notices, hold default, sorry, invalid option, numbers 1–20
voice-media/{accountId}/original/{mediaId}.{ext}
voice-media/{accountId}/play/{mediaId}.mp3
voice-media/generated/{hash}.mp3                      # shared cache for generated voice
voice-recordings/{accountId}/{yyyy}/{mm}/{callId}.mp3 # retention applies

6. Knowledge base (Phase 3)

  • Stored in the engine per account (business facts for call handling; not user data). Edited in Textify, pushed with PUT /v1/accounts/{id}/knowledge.
  • Structure: 
    { "profile": { "name": "ABC Ltd", "about": "...", "location": "...", "hours": "..." },
  "services": [ { "name": "Kifurushi cha mwezi", "price": "TZS 25,000", "notes": "..." } ],
  "faqs": [ { "q": "...", "a": "..." } ],
  "departments": [ { "id": "dep_sales", "description": "New orders and prices" } ],
  "rules": { "never": ["Promise refunds"], "handoverWhen": ["Caller asks for a person", "Complaint"] },
  "documents": [ "kb_doc_1" ] }
    (Kifurushi cha mwezi = "Monthly package".)
  • v1 of AI: prompt builder puts the whole KB into the AI session instructions (small businesses fit easily).
  • Large KB: chunk documents into Postgres (kb_chunks, full-text or pgvector) and give the AI a search_knowledge tool. Still just Postgres.
  • Live data via the account's endpoints exposed as AI tools.
  • Improvement loop: the summary job extracts unanswered questions → kb.unanswered event → Textify suggests FAQ additions.

7. Data model (PostgreSQL)

client_apps (id uuid pk, name, api_key_hash, api_key_hash_next null,status,                       -- ACTIVE | SUSPENDED | REMOVED
             webhook_url, webhook_secret_hash, can_call_url, rate_limit_rps, features jsonb,
             suspended_reason null, created_at, removed_at null)
client_app_keys (id uuid pk, client_app_id fk, key_hash, prefix, created_at, last_used_at,
                 last_used_ip, revoked_at null)                 -- ≥1 active; 2 keys forduring rotation
webhook_url,audit_log webhook_secret,(id can_call_url,bigserial created_at)pk, actor, action, target, data jsonb, ip, at)

accounts (id uuid pk, client_app_id fk, external_ref, status, default_language default 'sw',
          retention_days default 90, features jsonb, created_at, unique(client_app_id, external_ref))

phone_numbers (id uuid pk, e164 unique, provider default 'AT', type, status,
               client_app_id fk null,                          -- REGULARpool |owner TOLL_FREE(null status,= global pool)
               account_id fk null,
               label, flowhours jsonb, flow_versioncurrent_flow_version_id fk null, updated_at)

flows (id uuid pk, account_id fk, kind,                       -- NUMBER | CAMPAIGN
       owner_id uuid, created_at)
flow_versions (id uuid pk, flow_id fk, version int, updated_at)definition jsonb, created_at,
               premium_chars int default 0, unique(flow_id, version))

media_assets (id uuid pk, account_id fk null,                 -- null = system / shared generated
              kind,                                           -- UPLOAD | GENERATED | SYSTEM
              name, original_key null, play_key, text_hash null, tier null, lang, voice,
              mime, size_bytes, duration_sec, status, error null, created_at)

agents (id uuid pk, account_id fk, external_ref, display_name, status, last_seen_at,
        unique(account_id, external_ref))
ring_groupsdepartments (id uuid pk, account_id fk, name, strategy)strategy, --wrap_up_sec ALL_AT_ONCEdefault |0)
ROUND_ROBIN
ring_group_membersdepartment_members (ring_group_iddepartment_id fk, agent_id fk, position, pk(ring_group_id,department_id, agent_id))

endpoints (id uuid pk, account_id fk, name, url, secret, active bool, created_at)

calls (id uuid pk, account_id fk, number_id fk, campaign_id fk null, recipient_id fk null,
       flow_version_id fk, direction, from_e164, to_e164, at_session_id, livekit_room,
       status, handled_by, agent_id null, current_node,department_id null, context jsonb,
       started_at, answered_at, ended_at, end_reason,
       pstn_seconds, trunk_seconds, ai_seconds, wait_seconds, at_amount numeric, at_currency, dtmf_path text,
       ai_usage jsonb null)
call_events (id bigserial pk, call_id fk, type, node_id null, at, data jsonb)   -- trail

campaigns (id uuid pk, account_id fk, from_number_id fk, flow_version_id fk, status,             -- DRAFT|RUNNING|PAUSED|DONE|CANCELLED
           audio_url, tts_text, keys jsonb,
           window jsonb, cps int, max_concurrent int, max_retries int, retry_delay_sec int,
           starts_at, premium_chars_estimate int, created_at, completed_at)completed_at, pause_reason null)
campaign_recipients (id bigserial pk, campaign_id fk, e164, vars jsonbjsonb, media_id fk null,
                     status, attempts int,attempts, next_attempt_at, last_call_id fk null, answered_seconds,
                     keypress,keys text, index(campaign_id, status, next_attempt_at))
opt_outs (account_id fk, e164, source, created_at, pk(account_id, e164))

knowledge_bases (account_id pk fk, data jsonb, updated_at)                     -- Ph 3
kb_chunks (id bigserial pk, account_id fk, doc_id, text, tsv tsvector, embedding vector null)  -- later

recordings (id uuid pk, call_id fk unique, kind,                              -- CALL | VOICEMAIL
            source_url, storage_key, duration_sec, size_bytes, status, delete_after)
transcripts (call_id pk fk, status, language, text, segments jsonb, summary jsonb, model, created_at)

jobs (id bigserial pk, type, ref_id, status, attempts, run_after, last_error)
      -- DOWNLOAD|TRANSCRIBE|SUMMARIZE|…TRANSCODE | TTS | DOWNLOAD | TRANSCRIBE | SUMMARIZE | RETENTION
outbox_events (id uuid pk, client_app_id fk, type, payload jsonb, attempts, next_attempt_at,
               delivered_at null, last_error)
endpoint_deliveries (id uuid pk, endpoint_id fk, call_id fk, node_id, payload jsonb, mode,
                     attempts, next_attempt_at, status, response_code null, last_error)

Redis: presence:{accountId}:{agentRef}, call:{id}:agent (claim)claim, SET NX), wait:{groupId}departmentId} (sorted setset: ofcallId waitingby callIds)wait start), rr:{groupId}departmentId}, campaign:{id}:active, (concurrencyflow/number counter).config cache invalidation channel.

8. Security (service authentication)

TrafficMethod
Textify → engineAPI key per client app (hashed; 2 active keys for rotation; scoped to its accounts; rate-limited)
Engine → TextifyHMAC-SHA256 signed webhooks + timestamp (reject > 5 min)
Engine → customer endpointsHMAC with the endpoint's secret, idempotency key, HTTPS only, SSRF guard (block private/loopback/link-local/metadata IPs after DNS resolution), 3 s timeout
AT → engineUnguessable secret in path + AT IP allowlist + sanity checks (known number / call) [VERIFY AT IPs, any signature]
LiveKit → engineSigned JWT (WebhookReceiver)
OpenAI → engine (Ph 3)Webhook signature + dedupe
AT → LiveKit SIPFirewall + inbound trunk allowed_addresses = AT IPs; unknown callId → reject
Browser → LiveKitShort-lived room tokens minted by engine, requested through Textify (human auth happens there)
AT → audio URLsHMAC-signed, expiring media URLs

No OAuth/Keycloak/mTLS between our services. Secrets in env only. Recordings in private buckets with signed URLs, retention job, recording notice. Keep CrowdSec/fail2ban.

8.1 Client apps: how a service gets onto the engine, uses it, and is stopped or removed

Any service that consumes the engine (Textify first; later JikoXpress, Mauzodukani, NexGate…) is a client app. The engine treats them all the same and keeps them fully isolated from each other.

Who registers a service: only the engine owner (QBIT SPARK ops). There's no self-signup, and client apps can never create, see or manage other client apps. Textify is simply the first client app registered.

How: CLI-first (voice-admin), no public ops API

  • The admin tool is a mode of the engine jar (java -jar voice-engine.jar admin …, wrapped as voice-admin), run on the Voice box over SSH. It talks directly to the engine database, so no ops endpoint exists on the internet.
  • SSH: key-only login, root login disabled, port 22 open only to the owner's IPs (ufw), CrowdSec on.
  • Optional later (web ops panel): the /ops/* API binds only to localhost or a WireGuard VPN interface, never the public domain, and requires an ops token stored only on the box (env), plus the admin IP allowlist.
$ voice-admin client-app create --name textify \
    --webhook-url https://portal.textify.africa/voice/events \
    --can-call-url https://portal.textify.africa/voice/can-call \
    --features inbound,outbound,campaigns --rate-limit 50
→ clientAppId:    ca_7f3…
→ API key:        vk_live_3f9a…   (shown ONCE)
→ Webhook secret: whsec_81c2…     (shown ONCE)

$ voice-admin numbers allocate +2557XXXXXXXX --client ca_7f3…
$ voice-admin client-app test-webhook ca_7f3…
$ voice-admin key rotate ca_7f3…          # new key; old one stays valid until revoked
$ voice-admin key revoke ca_7f3… key_12…
$ voice-admin client-app suspend ca_7f3… --reason "…" [--hard]
$ voice-admin client-app resume ca_7f3…
$ voice-admin client-app export ca_7f3…  # zip of calls, recordings index, flows
$ voice-admin client-app remove ca_7f3…  # grace period, then delete
$ voice-admin audit --client ca_7f3… --since 7d

Security throughout

StageProtection
Who can registerOnly someone with SSH access to the Voice box (owner's key + IP)
Creating credentialsRandom 256-bit secrets generated by the engine; only hashes stored; shown once in the terminal
Hand-overOwner pastes them directly into the consuming service's env / secret store on its own server. Never email, chat, tickets or git
Identifiable keysPrefix vk_live_ / vk_test_ + short key id, so a leaked key is easy to spot and revoke; full key never logged
Using the engineHTTPS only; key → exactly one client app; all data filtered by client_app_id; per-client rate limit + feature flags
Webhooks backSigned with that client's webhook secret; client verifies signature + timestamp
Leak / staff changekey rotate (both valid briefly) → key revoke: effective immediately
Stop / removesuspend (reversible) / remove (export → release numbers → revoke keys → delete after grace)
TraceabilityEvery admin command (who, when, from which IP) and each key's first/last use + IP in audit_log

No approval steps: protection comes from who can reach the CLI plus show-once, hashed, scoped, rotatable keys.

Lifecycle

StepWhat happensHow
1. RegisterCreate the client app: name, webhook_url, can_call_url, rate limit, allowed features (inbound, campaigns, AI)voice-admin client-app create … → returns clientAppId, API key and webhook secret, shown once (only hashes stored)
2. Give it numbersAssign AT numbers from the global pool to the client app's poolvoice-admin numbers allocate <e164> --client <id>
3. Client stores credentialsThe client keeps the API key + webhook secret in its own secrets/envClient side
4. VerifyClient calls GET /v1/ping (auth check); engine sends a signed test.ping webhook to check the client's receivervoice-admin client-app test-webhook <id>
5. ConsumeClient creates its accounts (POST /v1/accounts {externalRef}), assigns numbers from its pool to accounts, saves flows, adds agents, runs campaigns…/v1/** with Authorization: Bearer <key>
6. Rotate keyIssue a second key; both work; client switches; old key revokedvoice-admin key rotate <id> → voice-admin key revoke <id> <keyId>
7. Suspend (stop, reversible)API returns 403 CLIENT_SUSPENDED; running campaigns paused; no new outbound calls; webhooks held in the outbox (delivered after resume). Inbound calls keep working by default, so callers aren't hurt. Option hard: true also answers inbound with a system "service unavailable" messagevoice-admin client-app suspend <id> --reason … [--hard] · … resume <id>
8. Remove (permanent)Suspend → cancel campaigns → release numbers back to the global pool (AT callback config untouched; numbers reusable) → revoke keys → stop webhooks → optional export (calls, recordings, flows as a zip) → data deleted after a grace period (default 30 days)voice-admin client-app export <id> · … remove <id>

Isolation rules

  • Every /v1 request resolves to exactly one client app from the API key; every query is filtered by client_app_id. A key can never see another client app's accounts, numbers, calls or media.
  • external_ref is unique per client app (two apps can both have "tenant_1").
  • Numbers belong to one client app's pool at a time; inbound calls are routed by number → client app → account.
  • Each client app has its own webhook URL + secret, rate limit (e.g. 50 req/s) and feature flags.
  • An account inside a client app can also be suspended (PATCH /v1/accounts/{id} {status: SUSPENDED}): same effect as above, but for one tenant. Textify uses this when a customer stops paying.

Audit: every admin command and key use (first/last seen, IP) is logged in audit_log; voice-admin audit reads it.

9. Communication

NeedMechanism
Textify ⇄ engine commands/queriesREST
Engine → Textify factsWebhooks via Postgres outbox + @Scheduled dispatcher
Engine → customer systemswebhook node: async via endpoint_deliveries (retries), sync direct (Ph 2)
AT call controlCallbacks + XML; AT REST call API
Ring agentsLiveKit data messages (lobby rooms)
Presence, claims, waiting orders, concurrency, cache invalidationRedis
Background workPostgres jobs + workers (FOR UPDATE SKIP LOCKED)
Broadcast queuecampaign_recipients table + dispatcher

Not used: MQTT, Kafka, RabbitMQ. Add Redis Streams first only if several engine instances + high event volume demand it.

10. APIs

6.10.1 Client API (Textify → engine)

· Authorization: Bearer <client_api_keyapi key>, scoped· to that client's accounts. Basebase https://voice-api.textify.africa/v1 [VERIFY domain].

only
Area Endpoints
Accounts POST /accounts · PATCH /accounts/{id}
Numbers GET /numbers?status=UNASSIGNED · POST /numbers/{id}/assign · POST /numbers/{id}/release · PUT /numbers/{id}/hours · PUT /numbers/{id}/flow (validates)validate + new version) · GET /numbers/{id}/flow/versions · POST /numbers/{id}/flow/rollback {version} · POST /numbers/{id}/test-call {toE164}
Flow toolsPOST /flows/validate (dry run, returns errors + premium char estimate)
MediaPOST /accounts/{id}/media (multipart) · POST /accounts/{id}/media/generate {text, lang, voice, tier} (preview) · GET /accounts/{id}/media · GET /media/{id} · DELETE /media/{id} · GET /voices
Agents PUT /accounts/{id}/agents/{ref} · DELETE … · PUT …/status · POST …/lobby-token
Ring groupsDepartments POST/PUT/DELETE /accounts/{id}/ring-groups[departments[/{gid}did}]
Calls POST /calls/outbound · POST /calls/{id}/accept · `POST /calls/{id}/hangup · POST /calls/{id}/transfer · GET /calls?… · GET /calls/{id} · GET /calls/{id}/recording (signed URL) · GET /calls/{id}/transcripthold
Campaigns POST /campaigns · POST /campaigns/{id}/recipients (chunks)chunks ≤ 10k) · POST /campaigns/{id}/estimate · `POST /campaigns/{id}/start
Opt-outs GET/POST/DELETE /accounts/{id}/opt-outs
OpsEndpoints POST/PUT/DELETE /accounts/{id}/endpoints[/{eid}] · POST /endpoints/{eid}/test
Knowledge (Ph 3)PUT /accounts/{id}/knowledge · POST /accounts/{id}/knowledge/documents · GET /accounts/{id}/knowledge/unanswered
StatsGET /numbers/{id}/stats (per node) · GET /campaigns/{id}/stats
HealthGET /ping (auth check)
Ops: CLI voice-admin on the box (primary). Optional /ops HTTP API bound to localhost/VPN only, ops token + admin IPs POST /ops/client-apps · POST /numbersops/client-apps/{id}/keys (register· ATDELETE number)/ops/client-apps/{id}/keys/{keyId} · POST /ops/client-apps/{id}/test-webhook · `POST /ops/client-apps/{id}/suspend

6.10.2 Engine → Textify

  • Balance check:Balance: GET {can_call_url}?accountRef&direction&estimatedCalls? → {allowed, maxMinutes?}; (1.5 ss, timeout → allow.fail-open).
  • Webhooks (outbox, signed, retried 1m/5m/30m/2h… up≤ to24 24h,h, idempotent by event id):
Event Key dataData
call.started /· call.answered callId, accountRef, direction, from, to, number, handledBy, agentRefagentRef, department
call.ended status, usage {pstnSeconds, trunkSeconds, waitSeconds, aiSeconds, atAmount}, recorded, endReasonendReason, flowVersion
call.missed callId, from, to, numberLabelnumberLabel, department
call.action_triggeredcallId, node, endpoint, status
message.requestedcallId/campaignId, to, channel, template, params → Textify sends SMS/WhatsApp
voicemail.received callId, from, mediaId/recording
recording.ready /· summary.ready callId,callId (+ summary JSON)
campaign.progressmedia.ready · media.failed countsmediaId, byduration status/ error
tts.usageaccountRef, chars, tier, ref (flow version / campaign)
campaign.progress · campaign.recipient_results · campaign.paused · campaign.completed batchcounts of/ {e164,batches status,/ seconds,reason keypress,/ atAmount}totals
campaign.completedkb.unanswered (Ph 3)accountRef, questions[]
test.ping totalssent on request to verify the client's webhook receiver

6.10.3 WebhooksHooks into the engine

Endpoint FromVerification
POST /hooks/at/{secret}/voice and· /step/{callId}/{node} · /wait/{callId} ATsecret in path + AT IP allowlist + destinationNumber/callerNumber must match a known number/call [VERIFY AT IPs; any signature]
POST /hooks/livekit LiveKitsigned JWT (WebhookReceiver)
POST /hooks/openai (Ph 3) OpenAI
webhookGET signature/media/{id}.mp3?t=… +· dedupeGET /media/tts/{hash}.mp3?t=…AT / LiveKit Ingress

7. Communication choices

NeedMechanism
Textify ⇄ engine commands/queriesREST
Engine → Textify factsWebhooks via Postgres outbox + @Scheduled dispatcher
AT call controlAT callbacks + XML; AT REST call API
Ring agents in browserLiveKit data messages (lobby room)
Presence, claims, waiting lists, campaign concurrencyRedis
Background work (download, transcribe, summarize, retention)Postgres jobs table + @Scheduled workers (FOR UPDATE SKIP LOCKED)
Broadcast queuecampaign_recipients table + dispatcher

Not used in v1: MQTT (IoT protocol, and browsers are covered by LiveKit), Kafka (overkill), RabbitMQ. When to add one: several engine instances and very high event volume, or many consumers of the same events → Redis Streams first (already running Redis), RabbitMQ only if outgrown. The outbox/jobs tables make that a contained swap.

8. Security (service authentication)

  • In (Textify → engine): API key per client app; stored hashed; two active keys for rotation; scoped to the client's accounts; rate-limited.
  • Out (engine → Textify): X-Voice-Event-Id, X-Voice-Timestamp, X-Voice-Signature: sha256=HMAC(secret, ts + "." + body); Textify rejects bad signatures and anything > 5 min old.
  • AT callbacks: unguessable secret in the path, IP allowlist, sanity checks against our numbers/calls.
  • LiveKit / OpenAI webhooks: built-in signature verification.
  • SIP: firewall and LiveKit inbound trunk allowed_addresses = AT IPs only. Inbound trunk accepts any user part (our callId) only from AT IPs [VERIFY trunk config without a numbers filter]. Unknown callId → reject.
  • Browsers: only short-lived room tokens, requested through Textify (where the human auth happens).
  • Recordings: private bucket, signed URLs (~5 min), retention job, recording notice.
  • No OAuth/Keycloak/mTLS between our services. Secrets only in env. Keep CrowdSec/fail2ban practice.

9.11. Infrastructure

9.11.1 Dedicated Voice box (Docker Compose; host networking for media)

Service Notes
caddy / nginx TLS for voice-api.… (engineAPI, APIhooks, +media hooks)URLs) and wss://voice-rtc.… (LiveKit)
livekit livekit/livekit-server, 7880 on localhost behind proxy
sip livekit/sip, 5060 + RTP 10000–20000, receives AT only
ingress livekit/ingress, agent-hold audio into rooms [VERIFY URL input for short audio]input]
redis shared by LiveKit + engine (separate DB index)
postgres engine DB (pgvector extension later for KB)
minioprivate buckets voice-media, voice-recordings; console on localhost
tts-basicself-hosted light TTS model (CPU), HTTP on localhost (if Basic passes the test)
voice-engine Spring Boot jar + ffmpeg
(Ph 3 optional)fallback) egress only if AT recording isn't enoughsufficient

Storage: S3-compatible private bucket (Linode Object Storage or MinIO). Start 4 vCPU / 8 GB;GB, measure. Choose the regionRegion with the lowest RTT to AT'sAT SIP servers.

9.11.2 Ports

Port Open to Purpose
22 adminowner IPs only (key-only login) SSHSSH, voice-admin CLI
80/443 world API, hooks, media, WSS
7881/tcp, 50000–60000/udp world WebRTC for (browser agentsagents)
5060 udp/tcp, 10000–20000/udp AT IPs only SIP + RTP from AT
7880, 6379, 5432, 9000/9001, tts-basic, 8080 localhost internal
(Ph 3) out 5061/tcp5061 + RTP to OpenAI CIDRs OpenAI AI over SIP

9.11.3 Env

AT_USERNAME / AT_API_KEY / AT_CALLBACK_SECRET / AT_IPS
LIVEKIT_URL=ws://127.0.0.1:7880LIVEKIT_URL / LIVEKIT_PUBLIC_WS_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET LIVEKIT_SIP_HOST=sip.voice.textify.africa/ LIVEKIT_SIP_HOST
DB_URL / DB_USER / DB_PASSWORD · REDIS_URL / REDIS_PASSWORD
S3_ENDPOINTMINIO_ENDPOINT / S3_BUCKETMINIO_ACCESS_KEY / S3_ACCESS_KEYMINIO_SECRET_KEY
MEDIA_URL_SECRET
TTS_BASIC_URL · TTS_PREMIUM_PROVIDER (openai|azure) / S3_SECRET_KEYkeys
OPENAI_API_KEY (Ph 2) · OPENAI_WEBHOOK_SECRET / OPENAI_SIP_URI (Ph 3)
AUDIO_HOLD_URL / AUDIO_CONSENT_URL / AUDIO_NO_BALANCE_URL

9.11.4 LiveKit bootstrap (once, idempotent)

  1. Inbound trunk at: allowed_addresses = AT IPsIPs, (no number filter, since the user part is our callId).filter.
  2. Dispatch rule dispatch_rule_callee with, room_prefix: "call-", so→ sip:{callId}@… →⇒ room call-{callId}. [VERIFY] callee rule uses the SIP user part..
  3. (Ph 3) Outbound trunk openai (TLS + SRTP).

10.12. Observability & testing

  • Logs:JSON JSON,logs with callId, accountId, atSessionId, campaignId, on every line.node.
  • Metrics: active calls, waiting callers,callers + wait time, ring-to-accept, answer/missed rate, ring-to-accept time, callback latency,latency (p95 < 500 ms), campaign dials/sec and+ answer rate, TTS jobs + premium chars, endpoint delivery failures, outbox backlog, job failures.backlog.
  • Alerts: SIP down, callback errors > N%,errors, zero inbound in business hours, outbox backlog, campaign stalled.stalled, TTS failures, system-file fallback used.
  • PoC checklist = SPEC §10 Phase 0 steps 1–5.0.
  • Unit: flow validator/executorvalidator, every NodeExecutor (node → XML), routingwait loop, claim race, waitingring order,strategies, campaign dispatcher (window, cps, retries, window/cps/retries/opt-out), outboxSwahili retry,normalizer, signatures.signers, SSRF guard.
  • Integration: ATTestcontainers sandbox/real(Postgres, number,Redis, MinIO), WireMock (AT, Textify, endpoints, TTS, OpenAI), LiveKit dev,dev + browser test page, Textifyreal webhookAT mock.number.

11.13. Tech stack & tools (one place)

datamessages
Layer Tool Used for
Engine Java 21, Spring Boot 3.x (Web, Validation, DataJDBC/JPA, JPA/JDBC,Scheduling, Scheduling)Actuator) All engine logic
DB PostgreSQL 16 + Flyway (pgvector later) State, flows, outbox, jobs, campaign queuequeue, KB
Cache/locks Redis 7 Presence, claims, waiting lists, campaignorders, concurrency (shared with LiveKit)
Telephony Africa's Talking Voice API: (callbacks + XML (Say,XML: Play, GetDigits, Dial, Record, Reject), outboundDial; call API, recordings;API); optional africastalking-java SDK for REST calls (XML built by our own small builder) PSTN both directions
Media LiveKit server, LiveKit SIP, LiveKit Ingress (hold audio); JVM server SDKIngress; io.livekit:livekit-server; JVM SDK; LiveKit JS SDK in Textify's agentTextify UI Rooms, SIP from AT, browser agents,agents
Voice (TTS)Basic: self-hosted light model (e.g. Piper, CPU) [VERIFY licence/quality]; Premium: OpenAI TTS or Azure SpeechText → audio files
AI OpenAI REST (transcription,transcribe, summaries) from Java RestClientsummarize); Ph 3: OpenAI3 Realtime over SIP + Java java.net.http.WebSocket client Summaries;Summaries, live AI
Storage S3-compatible (Linode Object Storage / MinIO)MinIO via AWS SDK v2 Recordings,Media audiolibrary, promptsgenerated voice, recordings
Audio processingffmpegTranscode, normalize, trim
HTTP clients Spring RestClient (sync, virtual threads), java.net.http.WebSocket (Ph 3) AT, Textify, endpoints, TTS, OpenAI
Security Hashed API keys (hashed, BCrypt/SHA-256),keys, HMAC-SHA256 webhooks,SHA256, LiveKit WebhookReceiver, OpenAISSRF webhookguard, verify, ufw firewall,ufw, CrowdSec Service auth
Edge Caddy (or nginx) TLS, reverse proxy for API, hooks, WSS
Deploy Docker Compose on aCompose, dedicated VPS All servicesEverything
Observability JSONLogback logs (Logback), SpringJSON, Actuator + Micrometer (Prometheus format)Prometheus) Logs, metrics, healthmetrics
Testing JUnit 5, TestcontainersTestcontainers, (Postgres, Redis), WireMock (AT/Textify/OpenAI mocks),WireMock, browser test page Unit + integration
Not used AT Say, MQTT, Kafka, RabbitMQ, OAuth/Keycloak, PythonPython, browser extension Not needed (§7, Appendix A)

12.14. Repository layout

textify-voice/
├── docs/          SPEC.md, FLOWS.md, ARCHITECTURE.md
├── voice-engine/  Spring Boot (Java 21)
├── tts-basic/     self-hosted TTS sidecar (if Basic passes)
├── deploy/        docker-compose.yaml, livekit.yaml, sip, ingress, Caddyfile, firewall.sh, .env.example
├── audio/         hold, consent, no-balancesystem prompts (sw/en): notices, hold default, sorry, invalid option, numbers
└── agent-test/    minimal HTML page with LiveKit JS for the PoC (real UI lives in Textify)

13.15. Open items to verify

  1. AT: Dial to sip:{callId}@host,; IPs,IPs; codecs/DTMF,DTMF; billing, recording ofbilling; Dial-to-SIP,SIP recording; CPS/concurrency, outboundconcurrency; per-second vs per-minute,minute; multi-number call API,API; callback fields and signature (SPEC §11).
  2. AT: doesGetDigits wrapping Play (key interrupt, timeouts, callbacks per call) for the wait loop; Play fetch timeout/streaming; XML continuecontinuation after a Dial leg ends (for voicemail after timeout)?Dial.
  3. LiveKit: callee dispatch rule with arbitrary user part; inbound trunk without number filter; ingressIngress URL input for hold audio.
  4. LiveKitinput; JVM SDK (io.livekit:livekit-server) method namesnames.
    5. for
  5. TTS: tokens,Basic datacandidate messages,quality ingress,+ participants.commercial licence; Premium choice; Swahili normalizer coverage.
  6. Ph 3: LiveKit → OpenAI SIP interop (TLS/SRTP), OpenAIinterop; Realtime vs Live API,API; Swahili AI quality and+ cost.

Appendix A: Phase 3 AI details (Java-only)

  • LiveKit has noNo Java SDK thatjoins joinsLiveKit rooms as an audio participant, andparticipant; Spring AI / LangChain4j are text-only,only. soSo OpenAI joins the room as a SIP participant instead..
  • Engine: CreateSIPParticipant(outbound trunk "openai", to = OpenAI project SIP URI, header X-Voice-Call-Id) → OpenAI webhook realtime.call.incoming (or Live API live.transport.incoming) → match header to a call in AI_DIALING → POST /v1/realtime/calls/{id}/accept with instructions (KB prompt + node instructions; Swahili first,first; short answers,answers; never invent prices), voice, tools (transfer_to_human, take_message, end_call, endpoint tools, send_sms) → WebSocket wss://api.openai.com/v1/realtime?call_id=… for tool calls/transcripts/usage → handoff: ring department, agent joins, POST …/hangup afterAI handover.leg.
  • Firewall: out TCP 5061 to sip.api.openai.com; RTP with OpenAI media CIDRs 13.79.45.80/28, 23.98.140.64/28, 40.67.149.176/28, 40.83.204.240/28 (OpenAI SIP guide, Oct 2026; re-check).
  • Trade-off: OpenAI voices only. If Swahili voice isn't good enough → fallback: small Python LiveKit Agents worker with another TTS, calling engine internal endpoints; nonothing otherelse change.changes.
Back to top