---
title: Speech-to-Text with OpenAI (streaming)
description: Replace the browser speech engine with the OpenAI Realtime API for gap-free streaming dictation in Jodit.
keywords: speech to text, speech recognize, openai, realtime, transcription, dictation, voice input
---

# Speech-to-Text with OpenAI (streaming)

Jodit's [`speech-recognize`](https://github.com/xdan/jodit/tree/main/src/plugins/speech-recognize) plugin ships with the browser's native Web Speech engine, but it was designed to be pluggable: the `speechRecognize.api` option accepts **any** constructor that behaves like the native `SpeechRecognition` object. That lets you swap in a cloud engine. This example uses the [OpenAI Realtime transcription API](https://developers.openai.com/api/docs/guides/realtime-transcription) over a WebSocket for streaming recognition.

> **Why streaming?** The native engine restarts after every phrase and loses audio in the gap, so fast speech gets skipped. The adapter below keeps a single microphone capture + WebSocket open for the whole session and lets OpenAI's server-side voice-activity-detection segment the speech, so dictation stays continuous with no gaps.

Enter your OpenAI API key, pick a model and language, then press the **microphone** button in the toolbar and start talking:

`{example SpeechRecognizeOpenAI}`

> **Security:** the demo sends your API key directly from the browser (the only way to open a WebSocket without a backend). Never do this in production: mint a short-lived [ephemeral key](https://platform.openai.com/docs/api-reference/realtime-sessions) on your server, or proxy the WebSocket through your backend.

## How it works

The plugin's `RecognizeManager` only needs an object with `start()`, `stop()`, `abort()` and the `result` / `end` / `error` / `speechstart` events. We implement exactly that contract and translate OpenAI's streaming events into it:

| OpenAI Realtime event                                   | Plugin event             | Effect                            |
| ------------------------------------------------------- | ------------------------ | --------------------------------- |
| `input_audio_buffer.speech_started`                     | `speechstart`            | Button pulses, watchdog re-armed  |
| `conversation.item.input_audio_transcription.delta`     | `result` (interim)       | Live preview popup updates        |
| `conversation.item.input_audio_transcription.completed` | `result` (final) → `end` | Text is committed into the editor |
| `error`                                                 | `error`                  | Error message shown               |

The one trick: `RecognizeManager.restart()` calls `stop()` then `start()` after every committed phrase. Our adapter defers teardown by a tick, so a restart just cancels it and keeps the same microphone + socket alive. The connection is never rebuilt, so no words are dropped between phrases.

## Wiring it into Jodit

```typescript
import { createOpenAISpeechRecognition } from './openai-speech-recognition';

const editor = Jodit.make('#editor', {
	buttons: ['bold', 'italic', '|', 'speechRecognize'],
	speechRecognize: {
		continuous: true,
		interimResults: true,
		sound: false, // Jodit beeps on every result via a leaked AudioContext
		// `api` is a constructor: the plugin instantiates it with `new ()`.
		api: createOpenAISpeechRecognition({
			getApiKey: () => document.querySelector('#openai-key').value,
			getModel: () => 'gpt-4o-mini-transcribe',
			getLanguage: () => 'en-US'
		})
	}
});
```

## The adapter

A self-contained implementation you can copy into your project. It captures the microphone, downsamples to PCM16 / 24 kHz, streams it to OpenAI, and forwards transcription events back to the plugin.

```typescript
// GA transcription endpoint. `?intent=transcription` = dedicated transcription
// session (no `?model=` needed); the beta shape is avoided by not sending the
// `openai-beta.*` subprotocol.
const OPENAI_REALTIME_URL =
	'wss://api.openai.com/v1/realtime?intent=transcription';
const TARGET_SAMPLE_RATE = 24000;

export interface OpenAISpeechRecognitionOptions {
	getApiKey(): string;
	getModel(): string;
	getLanguage(): string;
	onStatus?(status: string): void;
	onError?(message: string): void;
}

function floatTo16BitPCM(input: Float32Array): Int16Array {
	const output = new Int16Array(input.length);
	for (let i = 0; i < input.length; i++) {
		const sample = Math.max(-1, Math.min(1, input[i]));
		output[i] = sample < 0 ? sample * 0x8000 : sample * 0x7fff;
	}
	return output;
}

function downsample(
	input: Float32Array,
	inputRate: number,
	outputRate: number
): Float32Array {
	if (outputRate >= inputRate) {
		return input;
	}
	const ratio = inputRate / outputRate;
	const newLength = Math.round(input.length / ratio);
	const result = new Float32Array(newLength);
	for (let i = 0; i < newLength; i++) {
		result[i] = input[Math.min(Math.round(i * ratio), input.length - 1)];
	}
	return result;
}

function int16ToBase64(int16: Int16Array): string {
	const bytes = new Uint8Array(int16.buffer);
	let binary = '';
	for (let i = 0; i < bytes.length; i++) {
		binary += String.fromCharCode(bytes[i]);
	}
	return btoa(binary);
}

function buildResult(transcript: string, isFinal: boolean) {
	const item = {
		isFinal,
		length: 1,
		item: () => ({ transcript })
	};
	return { resultIndex: 0, results: { length: 1, item: () => item } };
}

export function createOpenAISpeechRecognition(
	options: OpenAISpeechRecognitionOptions
) {
	return class OpenAISpeechRecognition {
		lang: string | undefined;
		interimResults = true;
		continuous = true;

		private readonly _listeners = new Map<
			string,
			Set<(e: unknown) => void>
		>();
		private _ws: WebSocket | null = null;
		private _stream: MediaStream | null = null;
		private _audioCtx: AudioContext | null = null;
		private _processor: ScriptProcessorNode | null = null;
		private _source: MediaStreamAudioSourceNode | null = null;
		private _zeroGain: GainNode | null = null;
		private _segment = '';
		private _running = false;
		private _teardownTimer: number | null = null;

		addEventListener(event: string, callback: (e: unknown) => void): void {
			let set = this._listeners.get(event);
			if (!set) {
				set = new Set();
				this._listeners.set(event, set);
			}
			set.add(callback);
		}

		removeEventListener(
			event: string,
			callback: (e: unknown) => void
		): void {
			this._listeners.get(event)?.delete(callback);
		}

		start(): void {
			// restart() = stop() + start(): cancel the deferred teardown instead
			// of reconnecting, so audio keeps flowing between phrases.
			if (this._teardownTimer !== null) {
				clearTimeout(this._teardownTimer);
				this._teardownTimer = null;
				return;
			}
			if (this._running) {
				return;
			}
			this._running = true;
			void this._init();
		}

		stop(): void {
			this._scheduleTeardown();
		}

		abort(): void {
			this._scheduleTeardown();
		}

		private _emit(event: string, payload: unknown): void {
			const set = this._listeners.get(event);
			if (!set) {
				return;
			}
			// Snapshot listeners before dispatch (like the DOM does): the manager's
			// `end` handler restarts recognition, detaching and re-attaching itself
			// on this Set; iterating the live Set would loop forever.
			Array.from(set).forEach(callback => callback(payload));
		}

		private _scheduleTeardown(): void {
			if (!this._running || this._teardownTimer !== null) {
				return;
			}
			this._teardownTimer = window.setTimeout(() => {
				this._teardownTimer = null;
				this._teardown();
			}, 0);
		}

		private async _init(): Promise<void> {
			// The key travels as a WebSocket subprotocol → no whitespace/separators.
			const apiKey = options.getApiKey().replace(/\s+/g, '');
			if (!apiKey) {
				this._fail('Enter your OpenAI API key first', 'not-allowed');
				return;
			}
			if (!/^sk-[A-Za-z0-9_-]+$/.test(apiKey)) {
				this._fail(
					'API key looks invalid — paste only the sk-… key',
					'not-allowed'
				);
				return;
			}
			try {
				options.onStatus?.('Requesting microphone…');
				this._stream = await navigator.mediaDevices.getUserMedia({
					audio: {
						channelCount: 1,
						echoCancellation: true,
						noiseSuppression: true
					}
				});
			} catch {
				this._fail('Microphone access denied', 'not-allowed');
				return;
			}
			if (!this._running) {
				this._stopTracks();
				return;
			}
			options.onStatus?.('Connecting to OpenAI…');
			try {
				this._connect(apiKey);
				await this._startAudio();
			} catch (e) {
				const message = e instanceof Error ? e.message : String(e);
				this._fail(`Failed to start: ${message}`, 'voice-unavailable');
			}
		}

		private _connect(apiKey: string): void {
			// Browsers can't set WS headers; the key goes via the documented
			// (insecure, demo-only) subprotocol. No `openai-beta.*` protocol:
			// the beta Realtime shape was removed, this is the GA `/v1/realtime`.
			const ws = new WebSocket(OPENAI_REALTIME_URL, [
				'realtime',
				`openai-insecure-api-key.${apiKey}`
			]);
			this._ws = ws;

			ws.addEventListener('open', () => {
				// GA transcription session: config nested under `audio.input`,
				// audio format is an object (not the old `'pcm16'` string).
				ws.send(
					JSON.stringify({
						type: 'session.update',
						session: {
							type: 'transcription',
							audio: {
								input: {
									format: {
										type: 'audio/pcm',
										rate: TARGET_SAMPLE_RATE
									},
									transcription: {
										model: options.getModel(),
										language: (
											options.getLanguage() || 'en'
										)
											.split('-')[0]
											.toLowerCase()
									},
									turn_detection: {
										type: 'server_vad',
										threshold: 0.5,
										prefix_padding_ms: 300,
										silence_duration_ms: 500
									}
								}
							}
						}
					})
				);
				options.onStatus?.('Listening…');
			});

			ws.addEventListener('message', event => this._onMessage(event));
			ws.addEventListener('error', () =>
				this._fail('OpenAI connection error', 'voice-unavailable')
			);
			ws.addEventListener('close', event => {
				// 1000 = our own teardown; anything else while running means a
				// failed handshake (bad key / unsupported intent / network).
				if (this._running && event.code !== 1000) {
					this._fail(
						event.reason ||
							`Connection closed (code ${event.code})`,
						'voice-unavailable'
					);
				}
			});
		}

		private _onMessage(event: MessageEvent): void {
			if (typeof event.data !== 'string') {
				return;
			}
			let data: {
				type: string;
				delta?: string;
				transcript?: string;
				error?: { message?: string; code?: string };
			};
			try {
				data = JSON.parse(event.data);
			} catch {
				return;
			}
			switch (data.type) {
				case 'input_audio_buffer.speech_started':
					this._emit('speechstart', {});
					break;
				case 'conversation.item.input_audio_transcription.delta':
					if (typeof data.delta === 'string' && data.delta) {
						this._segment += data.delta;
						this._emit('speechstart', {});
						this._emit('result', buildResult(this._segment, false));
					}
					break;
				case 'conversation.item.input_audio_transcription.completed': {
					const text =
						typeof data.transcript === 'string' && data.transcript
							? data.transcript
							: this._segment;
					if (text) {
						this._emit('result', buildResult(text, true));
						this._emit('end', {});
					}
					this._segment = '';
					break;
				}
				case 'error':
					console.error('[openai-speech] server error:', event.data);
					this._fail(
						data.error?.message ??
							data.error?.code ??
							'OpenAI error',
						'voice-unavailable'
					);
					break;
			}
		}

		private async _startAudio(): Promise<void> {
			const stream = this._stream;
			if (!stream) {
				return;
			}
			const ctx = new AudioContext();
			// Browsers often start the context `suspended` (autoplay policy);
			// without resuming, onaudioprocess never fires → no audio is sent
			// and the button just keeps pulsing with nothing happening.
			if (ctx.state === 'suspended') {
				await ctx.resume();
			}
			if (!this._running) {
				void ctx.close();
				return;
			}
			const source = ctx.createMediaStreamSource(stream);
			const processor = ctx.createScriptProcessor(4096, 1, 1);
			const zeroGain = ctx.createGain();
			zeroGain.gain.value = 0; // keep the graph running without playback
			processor.onaudioprocess = e => this._onAudio(e);
			source.connect(processor);
			processor.connect(zeroGain);
			zeroGain.connect(ctx.destination);
			this._audioCtx = ctx;
			this._source = source;
			this._processor = processor;
			this._zeroGain = zeroGain;
		}

		private _onAudio(e: AudioProcessingEvent): void {
			const ws = this._ws;
			if (!ws || ws.readyState !== WebSocket.OPEN) {
				return;
			}
			const input = e.inputBuffer.getChannelData(0);
			const pcm = floatTo16BitPCM(
				downsample(input, e.inputBuffer.sampleRate, TARGET_SAMPLE_RATE)
			);
			ws.send(
				JSON.stringify({
					type: 'input_audio_buffer.append',
					audio: int16ToBase64(pcm)
				})
			);
		}

		private _teardown(): void {
			this._running = false;
			options.onStatus?.('Stopped');
			if (this._processor) {
				this._processor.onaudioprocess = null;
				this._processor.disconnect();
				this._processor = null;
			}
			this._zeroGain?.disconnect();
			this._source?.disconnect();
			this._zeroGain = null;
			this._source = null;
			if (this._audioCtx) {
				void this._audioCtx.close();
				this._audioCtx = null;
			}
			this._stopTracks();
			this._ws?.close();
			this._ws = null;
			this._segment = '';
		}

		private _stopTracks(): void {
			this._stream?.getTracks().forEach(track => track.stop());
			this._stream = null;
		}

		private _fail(message: string, code: string): void {
			console.error(`[openai-speech] ${message} (${code})`);
			options.onError?.(message);
			this._emit('error', { error: code });
			this._scheduleTeardown();
		}
	};
}
```

## Notes

- **GA API shape:** this uses the GA Realtime API: `session.update` with `session.type: 'transcription'` over `wss://api.openai.com/v1/realtime?intent=transcription`. The old beta `transcription_session.update` message and the `openai-beta.realtime-v1` subprotocol are no longer supported.
- **Models:** `gpt-4o-mini-transcribe` (fast, cheap) and `gpt-4o-transcribe` (most accurate) support server VAD, so the session auto-commits each phrase and you get streaming `delta`/`completed` events with no extra code. `gpt-realtime-whisper` is the newer streaming model but expects manual `input_audio_buffer.commit` (turn detection off), so it needs a little more wiring than this example does.
- **Audio format:** the GA endpoint expects `{ type: 'audio/pcm', rate: 24000 }` (mono little-endian PCM16 at 24 kHz); the adapter downsamples from the browser's native rate (usually 44.1/48 kHz).
- **`sound: false`:** older Jodit builds play a confirmation beep on every result by creating a fresh `AudioContext` they never close, so over a continuous session these accumulate into a "roar". This is fixed in Jodit (the sound helper now reuses one context per editor and releases it on destruct); the option stays here only as a workaround until that fix reaches the published build.
- **Editor content:** start the editor with a real block (`<p>…</p>`), not a bare text node: inserting recognized text into an unwrapped editor can hang Jodit's block normalization.
- **Browser support:** uses `getUserMedia`, `AudioContext` and `WebSocket`, available in all modern browsers (HTTPS required for microphone access).
- **Voice commands** still work: the plugin's `commands` map runs on the final transcript, so "new line", "select all", etc. behave exactly as with the native engine.

## Official OpenAI documentation

- [Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription): the `session.update` transcription shape, models, and event flow used above.
- [Realtime and audio guide](https://developers.openai.com/api/docs/guides/realtime): connecting over WebSocket/WebRTC, audio buffers and turn detection.
- [Realtime client events](https://platform.openai.com/docs/api-reference/realtime-client-events): `session.update`, `input_audio_buffer.append`, `input_audio_buffer.commit`.
- [Realtime server events](https://platform.openai.com/docs/api-reference/realtime-server-events): `conversation.item.input_audio_transcription.delta` / `.completed`, `error`.
- [Ephemeral keys for client-side auth](https://platform.openai.com/docs/api-reference/realtime-sessions): the production-safe alternative to putting your API key in the browser.