GPT-4o can analyse images, documents, charts, and screenshots in detail. This guide covers sending images via URL and base64, practical use cases, and building an image analysis pipeline.
GPT-4o's vision capabilities go far beyond simple image captioning. It can read text from images (OCR), interpret charts and graphs, analyse screenshots, understand diagrams and schematics, compare multiple images, and reason about spatial relationships — all in a single API call.
Practical applications: automated document processing, screenshot-to-code conversion, chart data extraction, accessibility descriptions for UI elements, quality control in manufacturing, and medical image analysis for supported use cases.
The simplest approach is to reference a publicly accessible image URL in the message content: `messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}, {"type": "text", "text": "What are the values in this chart?"}]}]`.
Use `detail: 'high'` for images that require reading small text or fine details: `{"image_url": {"url": "...", "detail": "high"}}`. High detail mode tiles the image into 512×512 chunks and processes each — it costs more tokens but produces significantly better results for documents and dense charts.
For local files or private images, encode to base64: `import base64; with open('image.jpg', 'rb') as f: data = base64.b64encode(f.read()).decode()`. Then: `{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{data}"}}`.
Supported formats: JPEG, PNG, GIF, WebP. Maximum image size is 20MB. For screenshots and UI analysis, use PNG to preserve crisp text rendering — JPEG compression artifacts reduce OCR accuracy.
Prompt engineering matters for data extraction. Instead of 'What does this chart show?', use: 'This is a bar chart. List every bar with its exact label and approximate value. Format your response as a JSON array of {label, value} objects.'
For financial charts, add: 'Values are in millions of dollars. Include units in the value field. If a value is not clearly readable, indicate uncertainty with a range (e.g. 45-50M).' Specific, structured prompts produce machine-parseable output that generic prompts do not.
Send a UI screenshot and ask the model to generate the implementation: 'Here is a screenshot of a web form. Write the HTML and CSS to replicate this UI as closely as possible. Use Tailwind CSS classes. Include all visible form fields and the submit button.'
For better results, use high detail mode and include the tech stack context: 'We are using React with TypeScript and Tailwind CSS. Generate a React component.' GPT-4o produces surprisingly accurate UI code from screenshots, especially for clean, standard UI patterns.
Whisper is OpenAI's speech recognition model that supports 99 languages with near-human accuracy. This guide covers the Whisper API, running Whisper locally for privacy, and building a transcription pipeline for long recordings.
Read guideOpenAI's image generation models produce high-quality images from text descriptions. This guide covers the Images API, prompt writing for consistent results, using the edit endpoint, and integrating image generation into applications.
Read guideTraditional RAG ignores charts, diagrams, and images embedded in documents. Multimodal RAG retrieves and reasons over both text and visual content. This guide covers document parsing, image embedding, and hybrid retrieval.
Read guide