Skip to main content

Textify Voice: Architecture

Companion to SPEC.md (what and why). This file is how: components, boundaries,adapters, call flows,flows (inbound, outbound, broadcast), data model, APIs, events,auth, infrastructure,communication, security.infra. Stack: Textify (Go) +→ Voice ServiceEngine (Java 21, Spring Boot) → Africa's Talking + LiveKit (self-hosted) + OpenAI Realtime (over SIP) + Africa's Talking. No Python.Python, Anythingno markedmessage broker. [VERIFY] must= be checkedcheck against current provider docs or Africa's Talking before coding against it.

1. System overviewOverview

                          ┌────────────────────────────────────────┐
  CallerPhones (any TZ network) │        Africa's Talking (AT)          │
        network) ◄──PSTN─── PSTN ──────►│  TZ numbers · telcoinbound interconnectcallbacks · XML ·   │
                          │ outbound call API · recordings · SIP  │
                          └───────────────────┬───────────────────▲──────────┘
             voice callbacks (POST)│                   │ XML replies, call API (REST)
             + Dial to SIP         trunk│                   (5060 + RTP)
                                         ▼│
┌───────────────────────── Textify Voice box (dedicated) ───────┼─────────────────────────────┐
│                                  │
│  ┌─────────────┐  Redis  ┌────────────────┐   ┌───────────────┐                  │
│  │ livekit-sip │◄───────►│ livekit-server │◄─►│ livekit-egress│──► S3 storage    │
│  └──────┬──────┘         └───────┬────────┘   └───────────────┘   (recordings)   │
│         │                        │            ┌───────────────┐                  │
│         │                        │◄──────────►│livekit-ingress│ (hold/consent    │
│         │                        │            └───────────────┘  audio files)    │
│         │                        │ server API + webhooks                         │
│         │                        ▼                   │                             │
│   ┌──────────────────────┐  SIP  ┌────────────┐               │
│         │             │   VOICE SERVICE      │◄────►│ PostgreSQL │               │
│         │             │   (Java Spring Boot) │      └────────────┘               │
│         │             │  numbers · routing · │                                   │
│         │             │  calls · AI control ·│                                   │
│         │             │  recordings · usage  │                                   │
│         │             └───┬──────────┬───────┘                                   │
└─────────┼─────────────────┼──────────┼──────────────────────────────────────────┐              │
│   │ livekit-sip  │◄──────┤            VOICE ENGINE (Spring Boot)    │──► Postgres  │
│   └──────┬───────┘ (AT)  │ accounts · numbers · flows · agents ·    │──► Redis     │
│          │               │ calls · campaigns · recordings · events  │──► S3 store  │
│   ┌──────▼──────────┐    │                                          │              │
│   │ livekit-server  │◄──►│  TelephonyAdapter  MediaAdapter  AiAdapter│             │
│   │ + ingress (hold)│    └───────▲──────────────────┬───────────────┘              │
SIP (TLS 5061   │   REST +└──────▲──────────┘            │ REST (clientAPI token)key)   ▲ signed│ webhooks │  + SRTP)(HMAC)              │
WebSocket│└──────────┼───────────────────────┼──────────────────┼──────────────────────────────┘
           │ WebRTC                │                  ▼
   ▼ (control)▼                      │┌───────┴────────┐       ┌──────┴──────────────────────────┐        ┌──────────────────────────┴──┐     ┌─────────────────┐
   │ OpenAIAgent Realtimebrowser  │  ◄─app─►│ TEXTIFY (Go)                     │◄───►│ Agent browser        │ OpenAI  │◄─ AiAdapter
   │ joins(Textify theUI call room as a+  │       │ users · tenants · wallet ·       │        app │ (TextifyPh web UI,│
   │ SIP participant; webhooks  │  │ pricing · dashboard ·       2–3)│
   │ LiveKit JS SDK) │       │ +pricing control· toUI Voice Service │  │· SMS/WhatsApp follow-ups      │        └────────┬────────┘
   └────────────────────────────┘       └─────────────────────────────────┘              │ WebRTC
                                                                                ▼
                                                                         livekit-server

KeyThree idea:rules every phone call is one LiveKit room. The caller,hold the AIdesign and human agents are all participants in that room:together:

  • the caller arrives from AT over the SIP trunk;
  • the AI is OpenAI Realtime, which Spring Boot dials into the room over SIP and controls through OpenAI's call-control REST + WebSocket;
    1. human agents join from the browser over WebRTC.

Spring BootAT is the only "brain"road to the phone network, both directions. Inbound arrives as AT callbacks; outbound starts with AT's call API.

  • Talking to a person (or AI) happens in LiveKit. The engine's XML says Dial sip:{callId}@<livekit-sip-host> and the call enters room call-{callId}. NothingLiveKit elseonly hasever businessreceives logic.

    calls from AT.
  • The engine is blind. It knows accountId, agentRef, numbers and calls, never users, roles or money.

    • 1.1 Responsibilities (hard boundary)

    Component Owns MustNever NOT ownowns
    Textify (Go) Users, login, roles, tenants, partners/white-label,partners, wallet, pricing,prices, invoices,all dashboardUI UI,screens (flow editor, campaigns, dashboard), contact lists, SMS/WhatsApp follow-ups AnyTelephony telephony,/ LiveKit or/ OpenAI logic
    Voice Service (Spring Boot)Engine Client apps, accounts (opaque ids)opaque), numbersnumbers, + config,flows, agents (opaque refs), ring groups, callpresence, lifecycle,calls, AI call control (OpenAI accept/tools/hangup),campaigns, recordings, transcripts, summaries, usage events Users, passwords,auth roles,of plans,people, prices, money
    LiveKit (server + SIP + egress + ingress)AT Real-timePSTN, media,numbers, XML execution, outbound dialing, recordingAnything else
    LiveKitRooms, WebRTC to browsers, SIP termination,from rooms,AT, recordinghold toaudio, storage,(Ph playing3) audioAI files into roomsparticipant Business logic
    OpenAI Realtime Listening,Ph thinking2: and speaking during AI calls (speech-to-speech); transcriptiontranscribe + summariessummarize. afterPh calls3: live voice State, routing decisions
    Africa's TalkingNumbers, PSTN, SIP trunkAnything else

    Rule of thumb: if a feature mentions people, plans, roles or money, it belongs in Textify. If it mentions calls, audio, numbers or AI behaviour on a call, it belongs in the Voice Service.

    1.2 Why a separate Voice Service

    • Reusable: JikoXpress, Mauzodukani and NexGate can become clients later (each is a client_app with accounts).
    • Failure isolation: a portal outage or deploy doesn't drop calls, and the other way round.
    • Security: the box with SIP ports open holds no user credentials.
    • One language for the call logic: Java (team strength). Textify stays Go.

    2. ComponentInside detailsthe engine

    2.1 Voice Service (Java Spring Boot)

    Stack: Java 21, Spring Boot 3.x, PostgreSQL + Flyway, Redis (locks, presence, short-lived call state),Redis, @Scheduled jobs (no Kafka at this scale),jobs, virtual threadsthreads.

    2.1 Adapters (plain interfaces, one OpenAIimplementation WebSocketeach, perno activeplugin AIframework)

    interface TelephonyAdapter {          // impl: AfricasTalkingAdapter
  String render(List<Action> actions);              // → AT XML
  OutboundResult placeCall(String from, List<String> to);   // AT call isAPI
  fine).InputStream downloadRecording(String url);
}

interface MediaAdapter {              // impl: LiveKitAdapter
  String sipUriFor(UUID callId);                    // sip:{callId}@sip.voice…
  String roomToken(String room, String identity, boolean canPublish);
  void sendData(String room, Object message);       // ring / ring_cancel to lobbies
  void startHoldAudio(String room); void stopHoldAudio(String room);
  void deleteRoom(String room);
  // Ph 3: dialAi(room, callId), removeParticipant(room, identity)
}

interface AiAdapter {                 // impl: OpenAiAdapter
  Transcript transcribe(InputStream audio, String langHint);   // Ph 2
  Summary summarize(Transcript t);                              // Ph 2
  // Ph 3: acceptLiveCall(openAiCallId, SessionConfig), control WebSocket, hangup
}

    Swapping later (another telco, another AI/TTS) = a new class, nothing else changes.

    Libraries

    2.2
    • LiveKit server SDK for JVM: io.livekit:livekit-server (Kotlin, usable from Java). Room service, SIP service (trunks, dispatch rules, CreateSIPParticipant, TransferSIPParticipant), egress, ingress, AccessToken, WebhookReceiver. [VERIFY] exact class/method names in the current version.
    • OpenAI: plain HTTPS (RestClient) for call accept/reject/refer/hangup, transcription and summaries; Java WebSocket client (java.net.http.WebSocket or Spring WebSocketClient) for live call control.
    • AWS SDK v2 S3 client (S3-compatible storage) for recordings.

    Package layout

    Packages

    com.qbitspark.voice
├── api          // client-facingclient REST (/v1/...),** client-token(API-key authauth)
├── hooks        // inbound webhooks:/hooks/at/{secret}/**, /hooks/livekit, /hooks/openai
├── telephony    // AfricasTalkingAdapter, XML builder (signature-verified)Say, Play, GetDigits, Dial, Record, Reject)
├── livekitmedia        // SDK wrapper,LiveKitAdapter, tokens, trunk/dispatch bootstrap,lobby/ring, ingress (hold audio), egressaudio
├── ai           // OpenAIOpenAiAdapter
call├── control: accept + session config, WebSocket session handler,
│flows        // toolflow execution,JSON AImodel, usagevalidator, capture;executor post-call(node transcription→ + summaryXML)
├── calls        // call lifecyclelifecycle, state machine, call timeline
├── routing      // ring groups, agent presence, accept/waiting list, claim logic(first accept wins)
├── campaigns    // broadcast campaigns, dispatcher, retries, opt-outs
├── recording    // egressdownload start/stop,→ storage,S3, retention
├── jobs         // Postgres job table + @Scheduled workers (transcribe, summarize, cleanup)
├── events       // outbox table + dispatcher (signed webhookswebhook to client apps)dispatcher
├── accounts     // client apps, accounts, numbers, number config, agents
└── common
// errors, ids, time, config

    2.23 AIFlow onmodel calls:(stored OpenAIper Realtimenumber, overbuilt SIP,in controlledTextify's from JavaUI)

    Caller ──AT──► LiveKit room{
  "call-…"record": (caller = SIP participant)
                  ▲
                  │ (1) Spring Boot: CreateSIPParticipant(outbound trunktrue,
  "openai"hours": { "tz": "Africa/Dar_es_Salaam", │"open": sip_call_to"Mon-Fri =08:00-17:00, <OpenAISat project SIP URI>09:00-13:00",
             │                  header X-Voice-Call-Id = <callId>)
                  ▼
          OpenAI Realtime (SIP, TLS 5061 + SRTP)
                  │ (2) webhook realtime.call.incoming"closed": {call_id, sip_headers}"say": ──►"Tumefunga POSTkwa /hooks/openaisasa. │Acha (3)ujumbe.", Spring"next": Boot:"vm" POST} /v1/realtime/calls/{call_id}/accept},
  │"start": "greet",
  "nodes": {model,
    voice,"greet": instructions,{ tools,"type": transcription"play", on}"url": │ (4) Spring Boot: WebSocket wss:"https://api.openai.com/v1/realtime?call_id={call_id}
                  │        ← function calls, transcripts, usage
                  │        → tool results,…/karibu.mp3", "saynext": this""menu" responses},
    │"menu":  (5){ Spring"type": Boot:"menu", POST"say": /v1/realtime/calls/"Bonyeza 1 mauzo, 2 huduma kwa wateja",
               "options": {call_id}/hangup (AI"1": leaves;"sales", room"2": stays)"support" },
               "noInput": "support", "retries": 2 },
    "sales":   { "type": "ring", "group": "grp_sales",   "timeoutSec": 30, "noAnswer": "vm" },
    "support": { "type": "ring", "group": "grp_support", "timeoutSec": 60, "noAnswer": "vm" },
    "vm":    { "type": "voicemail", "say": "Acha ujumbe baada ya mlio", "maxSec": 60 }
  }
}

    (The Swahili prompts read: "We're closed right now. Leave a message." · "Press 1 for sales, 2 for customer service" · "Leave a message after the tone".)

    Node types: say, play, menu, ring, voicemail, hangup (Ph 3: ai). The engine validates on save (every next exists, no loops without input).

    2.4 Executing a flow with AT callbacks

    • Each number's Correlationvoice callback URL: pass our callId as a custom SIP header on CreateSIPParticipant; read it back from data.sip_headers in OpenAI'sAT webhook.points to https://voice-api…/hooks/at/{secret}/voice.
    • The executor finds the number → flow → current node, and returns XML for that node.
    • State travels in the callback URL of each action (e.g. GetDigits callbackUrl=/hooks/at/{secret}/step/{callId}/menu), plus the calls row. No session memory needed.
    • The final callback (isActive=0) carries duration, cost (amount, currencyCode), recordingUrl, hangup cause → close the call. [VERIFY] LiveKitexact supportsAT customfield headersnames.
    • XML responses must be fast (< 500 ms): number/flow config cached in memory, refreshed on outbound SIP participants and OpenAI forwards them. Treat headers as untrusted: only accept calls whose callId exists in our DB with status AI_DIALING.
    • Session config on accept (built from the number's config):
      • instructions: base receptionist rules (Swahili first, short spoken answers, never invent prices, offer a human when unsure) + business name + ai_instructions + "start by saying the recording notice and greeting: …".
      • voice: chosen after the Swahili test.
      • tools: transfer_to_human, take_message(name, phone, reason), end_call. (Phase 2: account-defined tools proxied to the client app.)
      • Input transcription on (for live transcript + summaries).
    • Tool handling (on the WebSocket, in Java):
      • transfer_to_human → routing rings agents (§3.2) → tool result "ringing" → AI tells the caller to hold → on accept, AI briefs the agent in one sentence → Spring Boot hangs up the OpenAI leg.
      • take_message → save to calls.message_taken → event call.message_taken.
      • end_call → wait for the AI to finish speaking → delete the LiveKit room.
    • AI usage: capture usage from OpenAI response events and the leg duration (participant_joined/left of identity ai) → calls.ai_seconds, calls.ai_usage.
    • OpenAI API variants: OpenAI documents both the Realtime API (/v1/realtime/calls/...) and a newer Live API (/v1/live/sessions/..., webhook live.transport.incoming). Pick one at build time [VERIFY]. The design is the same.
    • Requirements: LiveKit outbound trunk to OpenAI uses TLS (5061) + SRTP; OpenAI requires SRTP, and trunks must support Opus with SDES-SRTP. LiveKit SIP supports TLS/SRTP. Interop [VERIFY] in the PoC.
    • Trade-off: locked to OpenAI's realtime voices and turn detection. If OpenAI's Swahili speech isn't good enough → Appendix A.

    2.3 LiveKit (self-hosted, dedicated box)

    ServicePurpose
    livekit-serverRooms, WebRTC for browser agents, server API, webhooks to the Voice Service
    livekit-sipTerminates the AT trunk (caller = SIP participant); dials OpenAI over SIP
    livekit-egressAudio-only room recording → S3-compatible storage
    livekit-ingressPlays pre-recorded audio (consent notice, hold music, "no balance" message) into a room from a URL [VERIFY] URL input with audio-only files
    redisRequired by server, SIP, egress and ingress

    Telephony objects (created once at bootstrap, reused for every call):

    • Inbound trunk "at-inbound": our AT numbers + AT IPs only (numbers, allowed_addresses).
    • Dispatch rule: dispatch_rule_individual, room_prefix: "call-", no agent dispatch (Spring Boot reacts to webhooks instead).
    • Outbound trunk "at-outbound": AT SIP host + credentials, for calls to customers.
    • Outbound trunk "openai": OpenAI SIP host, TLS transport, SRTP on, for dialing the AI into rooms.

    SIP participant attributes used: sip.phoneNumber (caller), sip.trunkPhoneNumber (our dialed number), sip.callID. [VERIFY] names, and that they are included in LiveKit webhook participant info.

    LiveKit SIP limitations: no SIP REGISTER (AT must send calls to our SIP URI directly); no video over SIP; DTMF, cold transfer (REFER), warm transfer, TLS/SRTP and noise cancellation supported. AT is not on LiveKit's tested-provider list, so the PoC must prove compatibility.

    Room naming: the individual dispatch rule puts the caller's number in the room name (PII in logs). We can't control the To header on calls from AT, so this is accepted for Phase 1 with access-controlled logs.

    2.4 Browser agent client (inside Textify's web app)

    • LiveKit JS SDK. Room tokens are minted by the Voice Service and fetched through Textify's backend; the browser never holds Voice Service credentials.
    • Lobby room per agent: lobby-{accountId}-{agentRef}, connected (mic off) while the agent is available:
      • Presence: lobby participant_joined/left webhooks = online/offline, automatically.
      • Ringing: the Voice Service sends a data message {type:"ring", callId, caller, numberLabel} into the lobby → incoming-call UI + ringtone.
    • Accept → Textify backend → POST /v1/calls/{id}/accept → first accept wins → returns a call-room token → the browser joins with mic on.
    • Losers get 409 and a {type:"ring_cancel"} data message.change.

    3. Call flows

    3.1 Inbound,Inbound AI_FIRST→ (mainmenu flow)→ browser agent

    Caller   AT                     Engine                          LiveKit             (sip/server)Agent Voice Service (Java)               OpenAI          Textifybrowser
  │dial─►│── dialPOST /voice ──────►│ number→account→flow           │                      │
  │      │                      │ │──can-call? INVITE(Textify, ────►│1.5s, trunk ok → room call-…fail-open)                 │
  │      │
  │        │               │◄── webhookXML: participant_joinedSay (SIP,notice to=our+ number)Play greeting + GetDigits ───────►│                      │
  │ │"2" │                             │ lookup number → account     │                │
  │        │               │                             ►│── GETPOST can-call/step/{id}/menu ─────────────┼──(dtmfDigits=2) ─────────────►│ │node=support (ring)  │
  │      │◄── XML: Dial sip:{allowed:true}callId}@sip.voice… record=true ───│                      │
  │      │══ SIP INVITE (user part = callId) ════════════════════════► room call-{callId}
  │      │                      │◄─ webhook participant_joined ─│                      │
  │      │                      │ start hold audio (ingress) ──►│                      │
  │      │                      │ pick available agents ─ ring data msg ─► lobby ─────►│ popup
  │      │                      │◄──────────┼──── accept (via Textify) ──────────────────│
  │      │                      │ │ create callclaim (RINGING),Redis eventSET call.startedNX) ──►│→ │room │token │◄── CreateSIPParticipant(openai trunk, X-Voice-Call-Id) ───│                │
  │        │               │── SIP INVITE (TLS/SRTP) ────┼────────────────────────────►│
  │      │                      │ │stop │◄── webhook call.incoming ───│                │
  │        │               │                             │── accept {instructions,tools}►               │
  │        │               │                             │── open control WebSocket ──►│                │
  │        │               │                             │ start egress (if recording) │                │
  │◄═══════╪═══hold audio ═════╪═══ AI: recording notice + greeting, conversation ═════════│                │
  │        │               │                             │◄── tool call: end_call ─────│                │
  │        │               │◄── delete room ─────────────│                             │                │
  │        │◄── BYE ───────│── webhooks: participant_left, room_finished ─────────────►│◄═════ joins, talks ══│
  │◄═════╪══════════════════════╪═════════ audio ═══════════════╪══════════════════════│
  │ hangup                      │                               │                      │
  │      │── computePOST usage/voice (isActive=0: duration, amount, recordingUrl) ►│ close call → call.ended ─┼───────────────►│ deduct walletevents

    If

      can-call
    • Waiting: saysif no: play the "insufficient balance" audio via ingress → delete the room → call REJECTED_NO_BALANCE (no AI cost). If Textify doesn't answer within 1.5 s: allow (fail-open) and log it.

      3.2 AI → human handover (warm transfer)

      1. AI calls tool transfer_to_human (caller asked for a person,agent is upset, or the AI can't answer).
      2. Voice Service: find the number's ring group → available agents (Redis presence) → send ring data messages to their lobbies (ALL_AT_ONCE or ROUND_ROBIN) → tool result {"status":"ringing"}.
      3. AI tellsavailable, the caller "nakuunganishastays naon mhudumuhold sasain hivi"the room; when an agent becomes AVAILABLE ("I'mor connecting you tofinishes a staff member now").
      4. Agent accepts → claim (SET call:{id}:agent NX) → call-room token → agent joins.
      5. Voice Service sendscall), the AIengine arings "briefagain. theOrder agent= inoldest one sentence" instruction over the WebSocket → after it speaks → POST …/calls/{call_id}/hangup on OpenAI → AI leg leaves; caller and agent keep talking. AI seconds stop.
      6. Timeout (default 30 s) with no accept → tool result {"status":"no_agent"} → AI takes a message → call.message_taken.

      3.3 Inbound, HUMAN_ONLY

      1. Same start as 3.1 (webhook → number lookup → can-call →waiting call record)first (Redis sorted set per group).
      2. No AI dialed.Timeout Voice Service starts ingress playing consent + hold audio into the room, and rings agents.
      3. Agent accepts → stop ingress → agent joins.
      4. Timeout → noAnswerActionnoAnswer:
          node.
        • AI: dial OpenAI intoSince the roomcall (asis now in 3.1)
          • LiveKit,
        • VOICEMAIL:voicemail playthere = the engine plays the prompt via ingress,ingress recordand records via egressegress, or simpler: remove the SIP participant so AT's Dial ends, then AT continues with the next XML (PhaseRecord) 2)on the Dial's callback. [VERIFY] AT continues the flow after Dial ends (prefer this: keeps voicemail on AT).
        • HANGUPMissed: deletecaller thehangs roomup →or MISSED
          • nobody
        handled
      5. MISSEDit → call.missed event → Textify sends the SMS/WhatsApp follow-up..

    3.4 Inbound, AI_ONLY

    Same as 3.1, but transfer_to_human isn't offered; the AI takes messages instead.

    3.52 Outbound / click-to-call (agent → customer)

    Agent browserclicks number → Textify → POST /v1/calls/outbound {accountId, numberId,fromNumberId, toE164,to, agentRef}
Voice Service:Engine:
  1. GET can-call (direction=call(OUTBOUND)
  2. create call + room out-call-{callId}
  3.; return agent token → browser joins (hears ringback)ringback 4.tone CreateSIPParticipant(at-outbound,from sip_call_to=toE164,page)
  sip_number=our3. number,TelephonyAdapter.placeCall(from, room,to) wait_until_answered)→ AT ringsdials the customercustomer; store AT sessionId → answercallId
Customer answers → AT callback (direction=Outbound, sessionId) → engine finds callId
  → XML: Dial sip:{callId}@sip.voice…  → customer joins the agent's room → talk
End → room_finishedfinal callback → call.ended (pstnSeconds outbound, trunkSeconds)

    3.63 TransferBroadcast to an external mobile (fallback, off by default)

    • Preferred: TransferSIPParticipant (SIP REFER) of the caller to tel:+255…. [VERIFY] whether AT honours REFER.
    • Alternative: CreateSIPParticipant dials the mobile into the same room (extra outbound leg, externalTransferSeconds).

    3.7 Recording + summary pipelinecampaign

    callTextify answered→ ──►POST start/v1/campaigns RoomComposite{accountId, egressfromNumberId, audioUrl|ttsText, keys:{1:"grp_sales",9:"optout"},
                              window, cps, maxConcurrent, retries} + recipients (audio_only)chunks of ≤10k)
Dispatcher (@Scheduled every 1s, per campaign):
  - inside calling window? not paused? concurrent < max?
  - SELECT … FROM campaign_recipients WHERE status='PENDING' AND next_attempt_at<=now()
      ORDER BY id LIMIT :cps FOR UPDATE SKIP LOCKED
  - skip opt-outs → mark OPTED_OUT
  - placeCall(from, batch) → mark DIALING, store AT sessionIds
Recipient answers → AT callback → XML: Play audio + GetDigits(keys)
  - "1" → Dial sip:{callId}@livekit → ring group (same as 3.1)
  - "9" → add to opt_outs → Say "Umeondolewa" ("You've been removed") → hangup
Final callback → recipient ANSWERED / NO_ANSWER / BUSY / FAILED (+seconds, amount)
  - NO_ANSWER/BUSY & attempts < max → PENDING with next_attempt_at = now + retryDelay
Events: campaign.progress every 30s (counts) · campaign.recipient_results batches · campaign.completed
Textify: SMS fallback to NO_ANSWER numbers; pauses campaign (POST /pause) if wallet low

    The campaign_recipients table is the queue. No broker needed.

    3.4 Recording → transcript → summary (Phase 2)

    final callback recordingUrl → job DOWNLOAD → S3 s3://recordings/rec/{accountId}/{callId}.oggmp3 room_finished→ ──►recording.ready
  egress ends ──► webhook egress_ended ──► recordings row READY
                                   │
                                   ▼ (@Scheduled→ job picks pending)
                 transcript: AI part already captured live from the OpenAI WebSocket;
                 full callTRANSCRIBE (incl.OpenAI, human agent part)sw/en) → OpenAIjob transcriptionSUMMARIZE (sw/en)OpenAI, │JSON) ▼
                 OpenAI chat (fixed prompt, JSON output) ──► summary
                                   │
                                   ▼
                 transcripts row DONE ──► event→ summary.ready
──► Textify
daily retention job ──►RETENTION → delete objects + rows where delete_after < now()

    Recording comes from AT (Dial record="true"), so no LiveKit egress service is needed. [VERIFY] AT records the Dial-to-SIP leg; fallback = LiveKit egress.

    Summary JSON:

    {summary, "summary":intent, "Mtejaresolved, aliuliza bei ya kifurushi cha mwezi na akaomba apigiwe kesho.",
  "intent": "pricing_inquiry",
  "resolved": false,
  "followUps": ["Call back tomorrow morning with monthly package price"followUps[], "sentiment":sentiment, "neutral",topic, "topic": "sales",
  "language": "sw"
}
language}

    (The example summary says: "The customer asked the price of the monthly package and asked to be called back tomorrow.").

    3.85 AI (Phase 3), same pattern

    Flow node ai (or noAnswer: ai, or campaign key 1: ai) → call is dialed into LiveKit as usual → engine dials OpenAI Realtime into the room over SIP (LiveKit outbound trunk, TLS 5061 + SRTP) → OpenAI webhook → engine accepts with instructions/tools → controls via WebSocket from Java → transfer_to_human rings agents → agent joins → engine hangs up the AI leg. Details kept in Appendix A.

    3.6 Call state machine

    INBOUND:  RINGING ──►→ AI_DIALINGIN_FLOW ──►→ WAITING_AGENT → ANSWERED ──►→ COMPLETED
│OUTBOUND: │DIALING │──→ (noANSWERED balance)→ ──►COMPLETED
any → MISSED | NO_ANSWER | BUSY | FAILED | REJECTED_NO_BALANCE
│──handled_by: (nobodyNONE handled| /FLOW_ONLY caller| hung up) ──► MISSED
   └── (SIP/system error) ──► FAILED      (AI_DIALING/ANSWERED can also → FAILED)

    handled_by: NONEAGENT | AI | AGENTAI_THEN_AGENT | AI_THEN_AGENT. If the OpenAI leg fails to connect within 5 s: fall back to ringing agents (HUMAN_ONLY path) and log AI_UNAVAILABLE.

    4. Browser agents

    • Agent UI lives in Textify's web app, using the LiveKit JS SDK.
    • Lobby room per agent: lobby-{accountId}-{agentRef}, connected with mic off while logged in.
      • Presence = lobby participant_joined/left webhooks (automatic online/offline) + explicit status from Textify.
      • Ringing = engine sendData(lobby, {type:"ring", callId, from, numberLabel, groupName}).
    • Accept → browser → Textify backend → POST /v1/calls/{id}/accept {agentRef} → engine claims (SET call:{id}:agent NX EX 60) → returns call-room token → browser joins with mic on. Losers: 409 + ring_cancel.
    • Browser never holds engine credentials; only short-lived room tokens (≈10 min to join, room-scoped).

    5. Data model (PostgreSQL)

    client_apps (
  id uuid pk, namename, text,api_key_hash, token_hashapi_key_hash_next text,null,   webhook_url-- text,2 webhook_secretkeys text,for can_call_urlrotation
             text,webhook_url, created_atwebhook_secret, timestamptzcan_call_url, )created_at)

accounts (
  id uuid pk, client_app_id uuid fk, external_refexternal_ref, text,       -- e.g. Textify tenant id
  status text,                                                 -- ACTIVE | SUSPENDEDstatus, default_language text default 'sw',
          recording_retention_days intretention_days default 90, created_atcreated_at, timestamptz,
  unique (unique(client_app_id, external_ref)
)

phone_numbers (
  id uuid pk, e164 text unique, provider text default 'AFRICAS_TALKING'AT', type text,type,          -- REGULAR | TOLL_FREE
               status text,                      -- UNASSIGNED | ACTIVE | RELEASEDstatus, account_id uuid fk null, labellabel, text,flow --jsonb, "Salesflow_version line"int, answer_mode text,                 -- AI_FIRST | AI_ONLY | HUMAN_ONLY
  greeting text,
  ai_instructions text,             -- business info, FAQs, tone
  ai_voice text null,               -- override default voice
  recording_enabled boolean default true,
  ring_group_id uuid fk null,
  ring_timeout_sec int default 30,
  no_answer_action text,            -- AI | VOICEMAIL | HANGUP
  external_transfer_e164 text null,
  updated_at timestamptz
)updated_at)

agents (
  id uuid pk, account_id uuid fk, external_refexternal_ref, text,display_name, --status, e.g.last_seen_at,
        Textify user id
  display_name text,
  status text,                      -- AVAILABLE | BUSY | OFFLINE (live presence also in Redis)
  last_seen_at timestamptz,
  unique (unique(account_id, external_ref)
)
ring_groups (id uuid pk, account_id uuid fk, namename, text, strategy text)strategy)                     -- ALL_AT_ONCE | ROUND_ROBIN
ring_group_members (ring_group_id uuid fk, agent_id uuid fk, positionposition, int,
                    primary key (pk(ring_group_id, agent_id))

calls (
  id uuid pk, account_id uuid fk, number_id uuid fk, directioncampaign_id text,                   -- INBOUND | OUTBOUND
  from_e164 text, to_e164 text,
  livekit_room text, sip_call_id text,
  openai_call_id textfk null,
       statusdirection, text,from_e164, handled_byto_e164, text,at_session_id, livekit_room,
       status, handled_by, agent_id uuid null, started_atcurrent_node,
       timestamptz,started_at, answered_atanswered_at, timestamptz,ended_at, ended_atend_reason,
       timestamptz,pstn_seconds, end_reasontrunk_seconds, ai_seconds,
       at_amount numeric, at_currency, dtmf_path text,
  inbound_seconds int default 0, outbound_seconds int default 0,
  trunk_seconds int default 0, ai_seconds int default 0,
  external_transfer_seconds int default 0,
  message_taken jsonb null,         -- {name, phone, reason} ai_usage jsonb null               -- model, token/audio counters (cost measurement)
)null)
call_events (id bigserial pk, call_id uuid fk, typetype, text, at timestamptz,at, data jsonb)

campaigns (id uuid pk, account_id fk, from_number_id fk, status,             -- timelineDRAFT|RUNNING|PAUSED|DONE|CANCELLED
           audio_url, tts_text, keys jsonb, window jsonb, cps int, max_concurrent int,
           max_retries int, retry_delay_sec int, starts_at, created_at, completed_at)
campaign_recipients (id bigserial pk, campaign_id fk, e164, vars jsonb null,
                     status, attempts int, next_attempt_at, last_call_id fk null,
                     answered_seconds, keypress, index(campaign_id, status, next_attempt_at))
opt_outs (account_id fk, e164, created_at, pk(account_id, e164))

recordings (
  id uuid pk, call_id uuid fk unique, egress_idsource_url, text,storage_key, storage_keyduration_sec, text,size_bytes,
            duration_secstatus, int, size_bytes bigint,
  status text,                      -- RECORDING | READY | FAILED | DELETED
  delete_after timestamptz
)delete_after)
transcripts (
  call_id uuid pk fk, statusstatus, text,  -- PENDING | DONE | FAILED
  language text, textlanguage, text, segments jsonb, summary jsonb, modelmodel, text,created_at)

created_atjobs timestamptz(id )bigserial pk, type, ref_id, status, attempts, run_after, last_error)   -- DOWNLOAD|TRANSCRIBE|SUMMARIZE|…
outbox_events (
  id uuid pk, client_app_id uuid fk, type text,type, payload jsonb, attemptsattempts, int default 0, next_attempt_at timestamptz,next_attempt_at,
               delivered_at timestamptz null, last_error text
)last_error)

    RedisRedis: keys:

    • presence:{accountId}:{agentRef}, → AVAILABLE|BUSY (TTL, refreshed by lobby webhooks)
    • call:{callId}id}:agent →(claim), claim lockwait:{groupId} (SETsorted NX,set TTL)
      • of
    • call:{callId}:aiwaiting →callIds), OpenAI call id + WebSocket owner node
    • rr:{ringGroupId}groupId}, campaign:{id}:active →(concurrency round-robin pointer
    counter).

    5.6. APIs

    5.6.1 Client-facingClient API (Textify → Voice Service)engine)

    Auth: Authorization: Bearer <client_app_tokenclient_api_key>. Every request is, scoped to that client app'client's accounts. JSON. Base https://voice-api.textify.africa/v1 [VERIFY domain].

    · URL)· +
    Method & pathArea PurposeEndpoints
    AccountsPOST /accounts Create/upsert· account {externalRef}
    PATCH /accounts/{id} Status, language, retention days
    Numbers GET /numbers?status=UNASSIGNED Numbers· available to assign
    POST /numbers/{id}/assign {accountId,· label}
    POST /numbers/{id}/release Unassign
    PUT /numbers/{id}/configflow answerMode,(validates) greeting,· aiInstructions,GET aiVoice, recordingEnabled, ringGroupId, ringTimeoutSec, noAnswerAction, externalTransferE164/numbers/{id}
    Agents PUT /accounts/{id}/agents/{agentRef}ref} Upsert agent· {displayName}DELETE … · PUT …/status · POST …/lobby-token
    DELETERing /accounts/{id}/agents/{agentRef}groupsRemove agent
    PUT /accounts/{id}/agents/{agentRef}/statusAVAILABLE / BUSY / OFFLINE
    POST /accounts/{id}/agents/{agentRef}/lobby-tokenLiveKit token for the agent's lobby
    POST/PUT/DELETE /accounts/{id}/ring-groups[/{gid}] Manage groups + members
    CallsPOST /calls/outbound · POST /calls/{id}/accept {agentRef}· → {roomToken, wsUrl} or 409
    POST /calls/{id}/hangup End· a call
    POST /calls/{id}/transfer {toAgentRef}· or {toE164}
    POST /calls/outbound{accountId, numberId, toE164, agentRef} → {callId, roomToken, wsUrl}
    GET /calls?accountId&from&to&status&page… Call· log
    GET /calls/{id} Detail· incl. timeline, message, summary
    GET /calls/{id}/recording Short-lived (signed URL
    GET /calls/{id}/transcript
    TranscriptCampaigns POST summary/campaigns · POST /campaigns/{id}/recipients (chunks) · `POST /campaigns/{id}/start
    Opt-outs

    Admin-GET/POST/DELETE /accounts/{id}/opt-outs Ops only (QBITPOST ops/client-apps token):· POST /numbers (register aAT numbernumber) bought from AT), POST /client-apps.

    5.6.2 Voice Service → Textify: balance check

    GET {can_call_url}?accountRef=…&direction=INBOUND|OUTBOUND → {allowed: bool, maxMinutes?: int}. Timeout 1.5 s → allow (fail-open), log it.

    5.3 Webhooks: Voice ServiceEngine → Textify

    • Balance check: POSTGET {webhook_url}can_call_url}?accountRef&direction&estimatedCalls? → {allowed, maxMinutes?}; headers1.5 X-Voice-Event-Id,s X-Voice-Event-Type,timeout X-Voice-Timestamp,→ X-Voice-Signature: sha256=HMAC(secret, timestamp + "." + body).allow.
    • Delivered via the outboxWebhooks with(outbox, exponentialsigned, backoffretried (1 m, 5 m, 30 m, 2 h, …1m/5m/30m/2h… up to 2424h, h).idempotent The receiver must be idempotent onby event id.id):
    Event PayloadKey highlightsdata
    call.started callId,/ accountExternalRef, direction, from, to, numberId
    call.answered callId, handledByaccountRef, (AI/AGENT),direction, agentRef?from, to, handledBy, agentRef
    call.ended callId, status, usage {inboundSeconds, outboundSeconds,pstnSeconds, trunkSeconds, aiSeconds, externalTransferSeconds,atAmount}, recorded},recorded, endReason
    call.missed callId, from, to, numberLabel.numberLabel → Textify sends the SMS/WhatsApp follow-up
    call.message_takenvoicemail.received callId, name,from, phone, reasonrecording
    recording.ready callId,/ durationSec
    summary.ready callId, (summary JSONJSON)
    campaign.progresscounts by status
    campaign.recipient_resultsbatch of {e164, status, seconds, keypress, atAmount}
    campaign.completedtotals

    Example

    6.3 call.ended:

    Webhooks 
    {
  "id": "evt_01J...", "type": "call.ended", "occurredAt": "2026-10-21T12:30:05Z",
  "data": {
    "callId": "3f2c...", "accountExternalRef": "tenant_123",
    "direction": "INBOUND", "from": "+2557XXXXXXXX", "to": "+2557YYYYYYYY",
    "status": "COMPLETED", "handledBy": "AI_THEN_AGENT", "agentRef": "user_88",
    "usage": { "inboundSeconds": 245, "outboundSeconds": 0, "trunkSeconds": 245,
               "aiSeconds": 70, "externalTransferSeconds": 0, "recorded": true },
    "endReason": "CALLER_HANGUP"
  }
}

    5.4 Inbound webhooks tointo the Voice Serviceengine

    POST /hooks/livekit: verified with LiveKit WebhookReceiver.

    ofleg
    LiveKit eventEndpoint ActionFromVerification
    participant_joinedPOST /hooks/at/{secret}/voice (SIP, roomand call-*/step/{callId}/{node}) StartAT secret anin inboundpath call:+ numberAT lookupIP →allowlist can-+ destinationNumber/callerNumber must match a known number/call →[VERIFY createAT callIPs; →any dial AI or ring agentssignature]
    participant_joinedPOST /hooks/livekit (identity ai) AILiveKit signed upJWT → (ANSWEREDWebhookReceiver, handled_by AI; start egress)
    participant_leftPOST /hooks/openai (identityPh ai)3) AI seconds end
    participant_joined (agent identity)OpenAI answered_at/handled_bywebhook AGENT;signature start+ egress if not running
    participant_left (SIP caller)Caller hung up → delete room
    participant_joined/left (lobby-*)Agent presence
    room_finishedClose call, compute durations, enqueue call.ended / call.missed
    egress_endedRecording READY/FAILED → enqueue transcriptiondedupe

    POST

    /hooks/openai:

    7. verifyCommunication OpenAI webhook signature, dedupe by event id.

    choices

    OpenAI eventNeed ActionMechanism
    Textify ⇄ engine commands/queriesREST
    Engine → Textify factsWebhooks via Postgres outbox + realtime.call.incoming@Scheduled dispatcher
    AT call controlAT callbacks + XML; AT REST call API
    Ring agents in browserLiveKit data messages (orlobby room)
    Presence, claims, waiting lists, campaign concurrencyRedis
    Background work (download, transcribe, summarize, retention)Postgres live.transport.incomingjobs table + @Scheduled workers (FOR UPDATE SKIP LOCKED)
    MatchBroadcast queueX-Voice-Call-Idcampaign_recipients headertable to+ a call in AI_DIALING → accept with session config → open control WebSocket. Unknown → rejectdispatcher

    DurationsNot used in v1: MQTT (IoT protocol, and browsers are alwayscovered computedby fromLiveKit), server-sideKafka webhook(overkill), timestamps.RabbitMQ. When to add one: several engine instances and very high event volume, or many consumers of the same events → Redis Streams first (already running Redis), RabbitMQ only if outgrown. The outbox/jobs tables make that a contained swap.

    6.8. Security (service authentication)

    • In (Textify → engine): API key per client app; stored hashed; two active keys for rotation; scoped to the client's accounts; rate-limited.
    • Out (engine → Textify): X-Voice-Event-Id, X-Voice-Timestamp, X-Voice-Signature: sha256=HMAC(secret, ts + "." + body); Textify rejects bad signatures and anything > 5 min old.
    • AT callbacks: unguessable secret in the path, IP allowlist, sanity checks against our numbers/calls.
    • LiveKit / OpenAI webhooks: built-in signature verification.
    • SIP: firewall and LiveKit inbound trunk allowed_addresses = AT IPs only. Inbound trunk accepts any user part (our callId) only from AT IPs [VERIFY trunk config without a numbers filter]. Unknown callId → reject.
    • Browsers: only short-lived room tokens, requested through Textify (where the human auth happens).
    • Recordings: private bucket, signed URLs (~5 min), retention job, recording notice.
    • No OAuth/Keycloak/mTLS between our services. Secrets only in env. Keep CrowdSec/fail2ban practice.

    9. Infrastructure

    6.9.1 BoxDedicated Voice box (Phase 0/1: single dedicated VPS, separate from NexGate)

    Docker Compose,Compose; host networking for media services (SIP/RTP/WebRTC need real ports).

    media) ,, ,shared
    ServiceImage / runtime Notes
    caddy (or/ nginx)reverse proxynginx TLS for voice-api.… (Springengine Boot,API incl.+ /hooks/*)hooks) and wss://voice-rtc.… (LiveKit 7880)LiveKit)
    livekit livekit/livekit-server bind7880 127.0.0.1:7880on localhost behind proxy; keys via LIVEKIT_KEYS envproxy
    sip livekit/sip config5060 via+ SIP_CONFIG_BODYRTP (api_key, api_secret, ws_url, redis, sip_port 5060, rtp_port 10000-10000–20000, use_external_ip);receives TLSAT enabled for the OpenAI trunk [VERIFY] SIP TLS config keys
    egresslivekit/egressaudio-only recordings to S3; CPU-heavy
    ingress livekit/ingress hold audio into rooms [VERIFY URL input for hold/consent/no-balanceshort audioaudio]
    redis redis:7 password,by localhostLiveKit + engine (separate DB index)
    postgres postgres:16Voice Serviceengine DB
    voice-serviceengine Spring Boot jar
    (JavaPh 21)3 optional) egress publiconly viaif proxyAT (APIrecording +isn't webhooks)enough

    Storage: private S3-compatible private bucket (e.g. Linode Object Storage or MinIO). SignedStart URLs4 only.vCPU Audio/ prompt files (consent, hold, no-balance) in a separate bucket/path, readable by ingress.

    Sizing (start, then measure): 4–8 vCPU,GB; 8–16 GB RAM.measure. Choose the region with the lowest RTT to Africa's Talking'AT's SIP servers (measure in the PoC); OpenAI SIP is GeoIP-routed (sip.api.openai.com, EU: sip-eu.api.openai.com).servers.

    6.9.2 Ports & firewall

    only API,
    Port ProtoDirection / openOpen to Purpose
    22 TCPin, admin IPs SSH
    80, 80/443 TCPin, world TLS issuance, API, webhooks,hooks, WSS
    78817881/tcp, 50000–60000/udp TCPin, world WebRTC TCPfor fallback (browser agents)
    50000–60000UDPin, worldWebRTC media (browser agents)agents
    5060 udp/tcp, 10000–20000/udp UDP+TCPin, AT IPs only SIP signalling+ RTP from AT
    10000–20000UDPAT IPs + OpenAI media CIDRs onlySIP RTP media
    5061TCPout, to sip.api.openai.comSIP over TLS to OpenAI
    7880, 6379, 5432, 8080 TCPlocalhost localhostinternal
    (Ph 3) out 5061/tcp + RTP to OpenAI CIDRs LiveKitOpenAI AI Redis,over Postgres, Spring BootSIP

    9.3 Env

    AT_USERNAME / AT_API_KEY / AT_CALLBACK_SECRET / AT_IPS
LIVEKIT_URL=ws://127.0.0.1:7880 / LIVEKIT_PUBLIC_WS_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET
LIVEKIT_SIP_HOST=sip.voice.textify.africa
DB_URL / DB_USER / DB_PASSWORD · REDIS_URL / REDIS_PASSWORD
S3_ENDPOINT / S3_BUCKET / S3_ACCESS_KEY / S3_SECRET_KEY
OPENAI_API_KEY (Ph 2) · OPENAI_WEBHOOK_SECRET / OPENAI_SIP_URI (Ph 3)
AUDIO_HOLD_URL / AUDIO_CONSENT_URL / AUDIO_NO_BALANCE_URL

    9.4 LiveKit bootstrap (once, idempotent)

    1. Inbound trunk at: allowed_addresses = AT IPs (no number filter, since the user part is our callId).
    2. Dispatch rule dispatch_rule_callee with room_prefix: "call-", so sip:{callId}@… → room call-{callId}. [VERIFY] callee rule uses the SIP user part.
    3. (Ph 3) Outbound trunk openai (TLS + SRTP).

    10. Observability & testing

    • Logs: JSON, with callId, accountId, atSessionId, campaignId on every line.
    • Metrics: active calls, waiting callers, answer/missed rate, ring-to-accept time, callback latency, campaign dials/sec and answer rate, outbox backlog, job failures.
    • Alerts: SIP down, callback errors > N%, zero inbound in business hours, outbox backlog, campaign stalled.
    • PoC checklist = SPEC §10 Phase 0 steps 1–5.
    • Unit: flow validator/executor (node → XML), routing claim race, waiting order, campaign dispatcher (window, cps, retries, opt-out), outbox retry, signatures.
    • Integration: AT sandbox/real number, LiveKit dev, browser test page, Textify webhook mock.

    11. Tech stack & tools (one place)

    LayerToolUsed for
    EngineJava 21, Spring Boot 3.x (Web, Validation, Data JPA/JDBC, Scheduling)All engine logic
    DBPostgreSQL 16 + FlywayState, outbox, jobs, campaign queue
    Cache/locksRedis 7Presence, claims, waiting lists, campaign concurrency (shared with LiveKit)
    TelephonyAfrica's Talking Voice API: callbacks + XML (Say, Play, GetDigits, Dial, Record, Reject), outbound call API, recordings; optional africastalking-java SDK for REST calls (XML built by our own small builder)PSTN both directions
    MediaLiveKit server, LiveKit SIP, LiveKit Ingress (hold audio); JVM server SDK io.livekit:livekit-server; LiveKit JS SDK in Textify's agent UIRooms, SIP from AT, browser agents, data messages
    AIOpenAI REST (transcription, summaries) from Java RestClient; Ph 3: OpenAI Realtime over SIP + Java WebSocket clientSummaries; live AI
    StorageS3-compatible (Linode Object Storage / MinIO) via AWS SDK v2Recordings, audio prompts
    HTTP clientsSpring RestClient (sync, virtual threads), java.net.http.WebSocket (Ph 3)AT, Textify, OpenAI
    SecurityAPI keys (hashed, BCrypt/SHA-256), HMAC-SHA256 webhooks, LiveKit WebhookReceiver, OpenAI webhook verify, ufw firewall, CrowdSecService auth
    EdgeCaddy (or nginx)TLS, reverse proxy for API, hooks, WSS
    DeployDocker Compose on a dedicated VPSAll services
    ObservabilityJSON logs (Logback), Spring Actuator + Micrometer (Prometheus format)Logs, metrics, health
    TestingJUnit 5, Testcontainers (Postgres, Redis), WireMock (AT/Textify/OpenAI mocks), browser test pageUnit + integration
    Not usedMQTT, Kafka, RabbitMQ, OAuth/Keycloak, PythonNot needed (§7, Appendix A)

    12. Repository layout

    textify-voice/
├── docs/          SPEC.md, ARCHITECTURE.md
├── voice-engine/  Spring Boot (Java 21)
├── deploy/        docker-compose.yaml, livekit.yaml, sip, ingress, Caddyfile, firewall.sh, .env.example
├── audio/         hold, consent, no-balance prompts (sw/en)
└── agent-test/    minimal HTML page with LiveKit JS for PoC (real UI lives in Textify)

    13. Open items to verify

    1. AT: Dial to sip:{callId}@host, IPs, codecs/DTMF, billing, recording of Dial-to-SIP, CPS/concurrency, outbound per-second vs per-minute, multi-number call API, callback fields and signature (SPEC §11).
    2. AT: does XML continue after a Dial leg ends (for voicemail after timeout)?
    3. LiveKit: callee dispatch rule with arbitrary user part; inbound trunk without number filter; ingress URL input for hold audio.
    4. LiveKit JVM SDK (io.livekit:livekit-server) method names for tokens, data messages, ingress, participants.
    5. Ph 3: LiveKit → OpenAI SIP interop (TLS/SRTP), OpenAI Realtime vs Live API, Swahili quality and cost.

    Appendix A: Phase 3 AI details (Java-only)

    • LiveKit has no Java SDK that joins rooms as an audio participant, and Spring AI / LangChain4j are text-only, so OpenAI joins the room as a SIP participant instead.
    • Engine: CreateSIPParticipant(outbound trunk "openai", to = OpenAI project SIP URI, header X-Voice-Call-Id) → OpenAI webhook realtime.call.incoming (or Live API live.transport.incoming) → match header to a call in AI_DIALING → POST /v1/realtime/calls/{id}/accept with instructions (Swahili first, short answers, never invent prices), voice, tools (transfer_to_human, take_message, end_call) → WebSocket wss://api.openai.com/v1/realtime?call_id=… for tool calls/transcripts/usage → POST …/hangup after handover.
    • Firewall: out TCP 5061 to sip.api.openai.com; RTP with OpenAI media CIDRs (from OpenAI's SIP guide, Oct 2026; re-check before go-live): 13.79.45.80/28, 23.98.140.64/28, 40.67.149.176/28, 40.83.204.240/28. WebRTC range and SIP RTP range must not overlap.

      6.3 Configuration (env)

      LIVEKIT_URL=ws://127.0.0.1:7880
LIVEKIT_PUBLIC_WS_URL=wss://voice-rtc.textify.africa
LIVEKIT_API_KEY / LIVEKIT_API_SECRET
REDIS_URL / REDIS_PASSWORD
DB_URL / DB_USER / DB_PASSWORD
S3_ENDPOINT / S3_BUCKET / S3_ACCESS_KEY / S3_SECRET_KEY
OPENAI_API_KEY
OPENAI_WEBHOOK_SECRET
OPENAI_SIP_URI                   # project SIP URI from OpenAI   [VERIFY format]
OPENAI_REALTIME_MODEL / OPENAI_DEFAULT_VOICE
AT_SIP_HOST / AT_SIP_USERNAME / AT_SIP_PASSWORD / AT_SIP_IPS
AUDIO_CONSENT_URL / AUDIO_HOLD_URL / AUDIO_NO_BALANCE_URL

      Client app tokens, webhook secrets and can-call URLs live in client_apps (tokens hashed).

      6.4 Bootstrap (one-time, idempotent on startup)

      1. Inbound trunk at-inbound (numbers + AT IPs).
      2. Dispatch rule (individual, call- prefix, no agent dispatch).
      3. Outbound trunk at-outbound (AT host + credentials).
      4. Outbound trunk openai (OpenAI SIP host,guide, TLS,Oct SRTP)2026; re-check).
      5. Configure theTrade-off: OpenAI project webhook → https://voice-api…/hooks/openai.

      Store the ids. Never create trunks/rules per call.

      7. Security

      • SIP exposed only to AT IPs (inbound) and OpenAI media CIDRs (firewall and LiveKit allowed_addresses).
      • OpenAI webhook signature verification + only accept OpenAI calls matching a call in AI_DIALING.
      • Client app → Voice Service: bearer token (hashed at rest), scoped to its accounts.
      • Voice Service → client app: HMAC-signed webhooks with timestamp (reject > 5 min old).
      • LiveKit webhooks: signature verified.
      • Browser gets only short-lived, room-scoped LiveKit tokens (~10 min to join).
      • OpenAI API key and SIP credentials only on the server.
      • Recordings: private bucket, signed URLs (~5 min), retention job, consent notice.
      • Secrets in envvoices only. Keep the CrowdSec/fail2ban practice from Textify's infra.

      8. Observability

      • Structured JSON logs with callId, room, accountId, openaiCallId on every line.
      • Metrics: active calls, active AI sessions, calls/min, answer rate, missed rate, AI handover rate, AI dial-in time (CreateSIPParticipant → accept), AI first-response latency, egress failures, webhook backlog, can-call timeouts.
      • Alerts: SIP service down, OpenAI dial-in failures > N%, zero inbound calls for X minutes in business hours, outbox backlog > N, egress failure rate > N%.

      9. Testing strategy

      • PoC checklist
        1. AT number → trunk → room created → participant_joined webhook received.
        2. Spring Boot dials OpenAI into the room → call.incoming webhook → accept → AI greets in Swahili → caller hears it.
        3. Tool call over the WebSocket (end_call) works → room deleted → durations correct.
        4. Handover: agent in browser joins, AI leg hangs up, caller stays.
      • Swahili quality test: 10–20 real callers (Swahili, English, mixed). Score understanding and voice. If the voice is weak → Appendix A.
      • Cost: log ai_usage per call; compute TZS per AI minute.
      • Unit: routing (ring strategies, claim race), state machine, outbox retry, signature verification, OpenAI tool dispatch.
      • Integration: LiveKit dev server + SIPp/softphone from an allowed IP; OpenAI in a test project; Textify webhook mock.

      10. Repository layout (suggested)

      textify-voice/
├── docs/                 SPEC.md, ARCHITECTURE.md
├── voice-service/        Spring Boot (Java 21)
├── deploy/               docker-compose.yaml, livekit.yaml, sip, egress.yaml, ingress.yaml, Caddyfile, firewall.sh, .env.example
├── telephony/            trunk + dispatch-rule JSON templates
└── audio/                consent / hold / no-balance prompt files (sw + en)

      11. Open items to verify before coding

      1. AT trunk behaviour (SPEC §11): external SIP URI, outbound auth, IPs, billing, REFER, codecs.
      2. LiveKit SIP → OpenAI SIP interop (TLS + SRTP, Opus/G.711), custom header pass-through, OpenAI project SIP URI format.
      3. OpenAI Realtime API vs Live API choice; webhook event names; usage fields.
      4. LiveKit JVM SDK method names (SIP participant create/transfer, egress, ingress, tokens).
      5. LiveKit SIP participant attributes present in webhook payloads.
      6. Ingress URL input for short audio prompts (latency to start playing).
      7. OpenAI Swahili quality + per-minute cost; choose a voice.
      8. Egress CPU usage for audio-only recordings on the chosen box.

      Appendix A: Fallback if OpenAI SIP fails the PoC (Python worker)

      Use only if (a) LiveKit ⇄ OpenAI SIP can't interoperate, or (b) OpenAI's Swahili voice isn't good enough and→ we need a different TTS provider.

      • Add afallback: small Python service using LiveKit Agents (livekit-agents[openai]),worker dispatched into each call room by the dispatch rule (room_config.agents).
      • It runs the voice loop (OpenAI Realtime, or STT + LLM +with another SwahiliTTS, TTS)calling and exposes the same three tools, which call the Voice Service over a localhost-onlyengine internal API (/internal/calls/start | handoff | message | end | ai-usage).
      • No database,endpoints; no businessother logic; about 150–300 lines. Spring Boot stays the brain; data model, client APIs and events above are unchanged.change.
    Back to top