Whisper is OpenAI's speech recognition model that supports 99 languages with near-human accuracy. This guide covers the Whisper API, running Whisper locally for privacy, and building a transcription pipeline for long recordings.
OpenAI's Whisper API is the simplest option: upload an audio file, get back a transcript. It costs $0.006 per minute, processes files up to 25MB, and handles most languages. For short recordings or low-volume use cases, it is the fastest path.
Running Whisper locally gives you: no cost per request, no file size limit, complete privacy, and offline operation. The trade-off is GPU memory and setup time. The `large-v3` model needs 10 GB VRAM for real-time speed; `medium` runs on 5 GB. For CPU inference, the `tiny` and `base` models are practical for non-real-time use.
Python: `from openai import OpenAI; client = OpenAI(); with open('audio.mp3', 'rb') as f: transcript = client.audio.transcriptions.create(model='whisper-1', file=f, language='en', response_format='text'); print(transcript)`.
For timestamped output useful in video editing or subtitles: set `response_format='verbose_json'`. The response includes word-level timestamps. For translation (any language to English): use `client.audio.translations.create(...)` with the same parameters.
Install: `pip install openai-whisper`. Transcribe: `import whisper; model = whisper.load_model('large-v3'); result = model.transcribe('audio.mp3'); print(result['text'])`. The model downloads automatically on first use. Use `faster-whisper` (CTranslate2-based) for 2–4× faster inference: `pip install faster-whisper`.
For GPU acceleration with faster-whisper: `from faster_whisper import WhisperModel; model = WhisperModel('large-v3', device='cuda', compute_type='float16'); segments, info = model.transcribe('audio.mp3'); transcript = ' '.join([s.text for s in segments])`.
The Whisper API has a 25MB limit. For longer files, split them into chunks: `pip install pydub`. Split on silence: `from pydub import AudioSegment, silence; audio = AudioSegment.from_file('long.mp3'); chunks = silence.split_on_silence(audio, min_silence_len=1000, silence_thresh=-40)`. Export each chunk and transcribe separately.
For local Whisper, long files are handled automatically — Whisper processes audio in 30-second windows internally. However, context doesn't carry between windows, which can affect accuracy at window boundaries. For best results on long recordings, use the `condition_on_previous_text=False` option to prevent error accumulation.
Raw Whisper transcripts lack punctuation accuracy for some models and may have hallucinations on silent segments. Post-process with an LLM: 'Correct any obvious transcription errors, add paragraph breaks, and fix punctuation in this transcript: [text]'. This takes under a second with gpt-4o-mini and significantly improves readability.
For speaker diarisation (who said what), combine Whisper with pyannote.audio: `pip install pyannote.audio`. pyannote identifies speaker segments which you then align with the Whisper word timestamps to label each sentence with its speaker.
GPT-4o can analyse images, documents, charts, and screenshots in detail. This guide covers sending images via URL and base64, practical use cases, and building an image analysis pipeline.
Read guideOpenAI's image generation models produce high-quality images from text descriptions. This guide covers the Images API, prompt writing for consistent results, using the edit endpoint, and integrating image generation into applications.
Read guideTraditional RAG ignores charts, diagrams, and images embedded in documents. Multimodal RAG retrieves and reasons over both text and visual content. This guide covers document parsing, image embedding, and hybrid retrieval.
Read guide