Parakeet Arabic / English ASR

Real-time automatic speech recognition via NVIDIA NIM parakeet-1-1b-rnnt-multilingual. Streaming WSS + REST transcription, automatic language detection, automatic punctuation, VAD, and speaker diarization.

1. Overview

The service transcribes spoken Arabic and English audio to text. Language is detected automatically from the audio stream — the client should not pin a language, otherwise auto-detection is disabled.

Backend model profile (do not change):

diarizer=sortformer, mode=str, type=default, vad=silero

type=default keeps automatic language detection enabled. Do not use type=prompt — that mode requires the client to specify the language up front.

2. Endpoints

3. Accepted languages

The model is multilingual and detects more than Arabic and English. A client-side whitelist must accept only:

See §10 Language whitelist for the implementation.

4. Audio format

The client must send audio in this exact format:

Convert any audio to the required format (ffmpeg)

copyffmpeg -i input.mp3 \
  -ar 16000 \
  -ac 1 \
  -sample_fmt s16 \
  output.wav

5. Authentication

The service is fronted by nginx with TLS termination. There is no API key or bearer token required from clients — access is controlled at the network layer. If you later need to add a key, insert it in nginx (e.g. map on $http_authorization).

6. Health check

copycurl https://t8.xhmi.com/v1/health/ready

Expected response:

{"object":"health.response","message":"ready","status":"ready"}

7. REST transcription (one-shot)

For pre-recorded files. Returns the full transcript after the audio is processed. Auto-detects language; the detected BCP-47 code is in the response so you can apply the whitelist.

curl

copycurl -X POST https://t8.xhmi.com/v1/audio/transcriptions \
  -F "file=@sample.wav" \
  -F "model=parakeet-1-1b-rnnt-multilingual" \
  -F "response_format=verbose_json"

Example JSON response (verbose)

{
  "task": "transcribe",
  "language": "ar-AR",
  "duration": 5.42,
  "text": "مرحبا، هذا اختبار للتعرف على الكلام.",
  "segments": [
    {
      "id": 0,
      "start": 0.0,
      "end": 5.42,
      "text": "مرحبا، هذا اختبار للتعرف على الكلام."
    }
  ]
}

Request fields

8. Streaming transcription (WSS)

For live audio from a microphone or stream. Open a WebSocket, send PCM16 chunks, and receive incremental transcript events.

Protocol

1. Open the WSS connection.
2. Server immediately sends conversation.created and session.created.
3. Optional: client sends session.update (defaults are fine — the session starts in multi auto-detect mode). Setting input_audio_transcription.language to a single code like en-US or ar-AR pins that language.
4. Client sends input_audio_buffer.append events with raw PCM16 bytes (base64).
5. Server streams back conversation.item.input_audio_transcription.delta partials as text grows.
6. When speech ends (VAD silence ≈ 1.2 s), the server sends conversation.item.input_audio_transcription.completed with the full utterance and the detected language.

Client → Server events

Server → Client events

9. Browser client (JavaScript)

copyconst ALLOWED = new Set(["ar-AR", "en-US", "en-GB"]);

const ws = new WebSocket(
  "wss://t8.xhmi.com/v1/realtime?intent=transcription"
);
ws.binaryType = "arraybuffer";

ws.onopen = () => {
  console.log("WSS connected");

  // Required: set the streaming model and language:"multi" (auto-detect).
  // Do NOT rely on defaults and do NOT pin a single language.
  ws.send(JSON.stringify({
    type: "transcription_session.update",
    session: {
      input_audio_format: "pcm16",
      input_audio_transcription: {
        model: "parakeet-rnnt-1.1b-unified-ml-cs-universal-multi-asr-streaming-silero-vad-sortformer",
        language: "multi"     // multilingual auto-detect (ar / en / …)
      },
      turn_detection: null   // use server default VAD (silero)
    }
  }));
};

ws.onmessage = (ev) => {
  const msg = JSON.parse(ev.data);
  switch (msg.type) {
    case "conversation.created":
      console.log("session:", msg.conversation.id);
      break;

    case "conversation.item.input_audio_transcription.delta":
      // partial text — language may be unknown until completed
      process.stdout.write(msg.delta || "");
      break;

    case "conversation.item.input_audio_transcription.completed":
      const lang = msg.language || msg.transcript?.language;
      const text = msg.transcript || "";
      if (!ALLOWED.has(lang)) {
        console.warn(`Rejected: detected language ${lang} not allowed`);
      } else {
        console.log(`
[${lang}] ${text}`);
      }
      break;

    case "error":
      console.error("server error:", msg.error);
      break;
  }
};

ws.onerror = (e) => console.error("WSS error:", e);
ws.onclose = (e) => console.log("WSS closed:", e.code, e.reason);

// ---- Feed PCM16 (16 kHz mono) chunks from Web Audio/MediaRecorder ----
function sendAudioChunk(int16Samples) {
  // int16Samples: Int16Array of PCM16 samples
  const bytes = new Uint8Array(int16Samples.buffer);
  let binary = "";
  for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]);
  const b64 = btoa(binary);
  ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: b64
  }));
}

Resample microphone audio to 16 kHz PCM16 in the browser

copyasync function micStream() {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { channelCount: 1, sampleRate: 16000, echoCancellation: true }
  });
  const ctx = new AudioContext({ sampleRate: 16000 });
  const src = ctx.createMediaStreamSource(stream);
  const bufSize = 4096;
  const node = ctx.createScriptProcessor(bufSize, 1, 1);
  src.connect(node);
  node.connect(ctx.destination);
  node.onaudioprocess = (e) => {
    const f32 = e.inputBuffer.getChannelData(0);
    const i16 = new Int16Array(f32.length);
    for (let i = 0; i < f32.length; i++) {
      let s = Math.max(-1, Math.min(1, f32[i]));
      i16[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
    }
    sendAudioChunk(i16);
  };
}

Live output & sentence segmentation

To get a live, sentence-by-sentence display, the browser client does two things:

1. Periodic commit (~every 0.9 s) while the user is talking, so the server keeps flushing an updated transcript in near real time.
2. Client-side silence detection (energy/RMS VAD): when the audio stays below a threshold for ~0.9 s after speech, the current line is finalized as a sentence and a fresh line begins.

Because deltas are cumulative, each new sentence is obtained by stripping the text already finalized (a prefix) from the latest delta — this yields a clean new sentence whether or not the server resets its context on the pause.

copyconst SPEECH_RMS = 0.015;   // energy above this = speech
const SILENCE_MS = 900;     // silence this long after speech = sentence end
let finalizedPrefix = "";   // full server text at the last finalize
let lastFullText = "";      // most recent cumulative delta
let hasSpeech = false, lastVoice = 0;

// In onaudioprocess, after computing rms of the chunk:
if (rms > SPEECH_RMS) { hasSpeech = true; lastVoice = performance.now(); }
else if (hasSpeech && performance.now() - lastVoice > SILENCE_MS) {
  hasSpeech = false;
  finalizeSentence();
}

// On each delta: show only the current sentence (strip finalized prefix)
function onDelta(fullText) {
  lastFullText = fullText;
  let part = fullText;
  if (finalizedPrefix && fullText.startsWith(finalizedPrefix))
    part = fullText.slice(finalizedPrefix.length);
  liveLine.textContent = part.trim();
}

// On a pause: keep the current line and start a fresh one
function finalizeSentence() {
  if (liveLine.textContent.trim()) appendSentence(liveLine.textContent.trim());
  finalizedPrefix = lastFullText;                 // everything so far is now done
  liveLine.textContent = "";
  ws.send(JSON.stringify({ type: "input_audio_buffer.commit" }));
}

10. Python client (streaming)

copyimport asyncio, json, base64, ssl
import websockets

ALLOWED_LANGUAGES = {"ar-AR", "en-US", "en-GB"}
URL = "wss://t8.xhmi.com/v1/realtime?intent=transcription"

async def stream_audio(pcm16_chunks):
    """
    pcm16_chunks: iterable of bytes objects, each containing raw PCM16
    (16 kHz, mono, signed 16-bit LE).
    """
    async with websockets.connect(URL) as ws:
        # Initial server message
        first = json.loads(await ws.recv())
        assert first["type"] == "conversation.created"
        print(f"[session] {first['conversation']['id']}")

        # Required session update: set the streaming model + language "multi"
        await ws.send(json.dumps({
            "type": "transcription_session.update",
            "session": {
                "input_audio_format": "pcm16",
                "input_audio_transcription": {
                    "model": "parakeet-rnnt-1.1b-unified-ml-cs-universal-multi-asr-streaming-silero-vad-sortformer",
                    "language": "multi"
                }
            }
        }))

        async def send_audio():
            for chunk in pcm16_chunks:
                payload = {
                    "type": "input_audio_buffer.append",
                    "audio": base64.b64encode(chunk).decode("ascii")
                }
                await ws.send(json.dumps(payload))
                await asyncio.sleep(0.1)  # ~100 ms cadence
            await ws.send(json.dumps({"type": "input_audio_buffer.commit"}))

        async def recv_results():
            while True:
                msg = json.loads(await ws.recv())
                t = msg.get("type")
                if t == "conversation.item.input_audio_transcription.delta":
                    print(msg.get("delta", ""), end="", flush=True)
                elif t == "conversation.item.input_audio_transcription.completed":
                    text = msg.get("transcript", "")
                    lang  = msg.get("language")
                    if lang not in ALLOWED_LANGUAGES:
                        print(f"
[REJECTED] detected {lang}, not in {sorted(ALLOWED_LANGUAGES)}")
                    else:
                        print(f"
[{lang}] {text}")
                    return
                elif t == "error":
                    print(f"
[ERROR] {msg.get('error')}")
                    return

        await asyncio.gather(send_audio(), recv_results())

# Example: stream a 16 kHz mono WAV's PCM body
def wav_pcm16_chunks(path, chunk_ms=100):
    import wave
    w = wave.open(path, "rb")
    assert w.getframerate() == 16000 and w.getnchannels() == 1
    frames_per_chunk = int(16000 * chunk_ms / 1000)
    while True:
        data = w.readframes(frames_per_chunk)
        if not data: break
        yield data

asyncio.run(stream_audio(wav_pcm16_chunks("sample.wav")))

Python — REST one-shot

copyimport requests

ALLOWED = {"ar-AR", "en-US", "en-GB"}

with open("sample.wav", "rb") as f:
    r = requests.post(
        "https://t8.xhmi.com/v1/audio/transcriptions",
        files={"file": f},
        data={
            "model": "parakeet-1-1b-rnnt-multilingual",
            "response_format": "verbose_json",
        },
    )
r.raise_for_status()
data = r.json()
lang = data.get("language")

if lang not in ALLOWED:
    raise SystemExit(f"unsupported_language: detected {lang}")
print(f"[{lang}] {data['text']}")

11. Language whitelist

The model emits more languages than Arabic/English. Always filter on the detected language returned by the server.

copyALLOWED_LANGUAGES = {"ar-AR", "en-US", "en-GB"}

def filter_result(transcript_event):
    lang = transcript_event.get("language")          # BCP-47 from server
    text = transcript_event.get("text") or transcript_event.get("transcript") or ""
    if lang not in ALLOWED_LANGUAGES:
        return {
            "error": "unsupported_language",
            "detected_language": lang,
            "allowed": sorted(ALLOWED_LANGUAGES),
        }
    return {"language": lang, "text": text}

12. Errors

13. Server metadata

copycurl https://t8.xhmi.com/v1/metadata

Returns model info, GPU, profile, version. Useful for support tickets.

14. Limits & timeouts