Beyond Text: AI That Processes Multiple Input Types
The AI systems that first reached mainstream awareness — the text-based chatbots and language models — operate on a single modality: text in, text out. The expanding frontier of AI development is multimodal systems: models that can process multiple input types simultaneously (text, images, audio, video, code) and generate responses that integrate understanding across these modalities. GPT-4V (Vision), Gemini (natively multimodal from the start), Claude’s image and document processing, and similar systems represent the current generation of widely accessible multimodal AI.
The practical significance of multimodality is in the use cases it enables that single-modality AI cannot serve: analyzing a photograph and answering questions about it, reading a chart and explaining its findings, watching a video and summarizing its content, transcribing spoken audio, describing what a piece of code is doing when shown a screenshot of it. These aren’t marginal capabilities — they unlock genuinely new applications for AI in workflows that involve non-text information.
Image Understanding: What Current Models Can and Can’t See
Current multimodal AI models can describe the content of images with impressive accuracy, answer specific questions about image content, identify objects and relationships in photographs, read text within images (OCR capability integrated with language understanding), and analyze charts, graphs, and diagrams to explain what they show. For tasks like ‘what are the main trends in this chart’ or ‘describe what’s happening in this photograph’ or ‘is there a stop sign in this image’, current models are reliable enough for practical use.
The failure modes worth knowing: current models hallucinate image content — describing elements that aren’t present in the image with the same confident fluency as accurately described elements. Counting specific objects in images is unreliable for more than a few obvious items. Precise spatial relationships (‘is the blue box to the left of the red circle’) are often incorrect. And the models can be systematically fooled by adversarial images or context — their image understanding is statistical pattern matching rather than geometric understanding.
Audio and Video: The Emerging Frontier
Audio processing in multimodal AI includes speech-to-text transcription (which has been a standalone AI application for years but is now integrated into larger multimodal systems) alongside more sophisticated audio understanding: identifying speakers in a multi-speaker recording, summarizing the content of a meeting recording, answering questions about a podcast episode. Whisper (OpenAI’s open-source transcription model) and its successors have set a high bar for transcription accuracy that’s now available in various integrated systems.
Video understanding is the least mature of the major modalities at the consumer level: processing video requires handling many frames with temporal relationships between them, which is computationally expensive and produces more inconsistent results than image processing. Current video AI can describe the general content of short video clips, answer questions about what happens in a video, and generate text descriptions or summaries. Reliable temporal understanding — ‘at what point in the video does X happen’ or ‘describe what changes between the beginning and end’ — is improving but not yet as reliable as static image understanding.
The Document and Chart Reading Use Case
One of the most immediately practical multimodal AI applications for business users is document and chart analysis: uploading a PDF report and asking questions about its content, uploading a screenshot of a spreadsheet chart and asking for an explanation of what it shows, or uploading a contract and asking for a summary of specific clauses. These use cases work well with current models because they leverage both the text-reading capability (OCR for text within images, PDF text extraction) and the visual understanding capability (chart interpretation, layout understanding).
The practical limitation: for consequential document analysis (legal contracts, financial filings, medical reports), AI output should be verified against the source document rather than trusted as a complete and accurate representation. The AI may miss nuanced language, misread specific numbers, or fail to flag the clause that matters most for the specific situation. The appropriate use is as a first-pass reading aid that directs human attention to relevant sections, not as a replacement for expert review of important documents.
Building Multimodal AI Into Products
For developers and businesses building products on multimodal AI, the APIs for multimodal models are now accessible through the same providers as text-only models. Anthropic’s Claude API accepts images and documents alongside text prompts; OpenAI’s API accepts images for GPT-4V and successors; Google’s Gemini API is natively multimodal. Building a product feature that analyzes user-uploaded images, transcribes audio, or processes document uploads is now an API integration task rather than a machine learning research project.
The practical considerations for multimodal API integration: images and documents consume significantly more tokens than equivalent text (a full-page image might use as many tokens as several paragraphs of text), which affects per-call cost at scale. The latency for processing images is higher than text-only responses. And the privacy implications of sending images and documents to API providers need evaluation against the data handling policies discussed in the Vol. 1 AI articles — images may contain sensitive information that the text description of a task wouldn’t.
