IntermediateИнженерия

OpenAI Audio & Vision: Whisper and GPT-4V

Official OpenAI examples for audio and vision: transcription with Whisper, prompting for better results, Realtime API, call summarization, and image tagging with GPT-4V.

3modules

5lessons

125 mintotal time

Developers working with audio and images via OpenAIaudience

Module 1

Transcription with Whisper

Audio pre-processing and transcript post-processing: silence trimming, segmentation, punctuation and domain-vocabulary correction.

Whisper: Audio Pre- and Post-Processing

Improve Whisper transcription quality by trimming leading silence, splitting long files into segments, and running GPT post-processing to add punctuation, fix financial terminology, and remove non-ASCII artefacts.

25 min

Module 2

Prompting and Transcription Methods

Fine-tuning Whisper through the prompt parameter, choosing the right transcription method (batch vs streaming vs Realtime), and comparing trade-offs.

Whisper Prompting: Better Transcription Output

Use the optional Whisper API prompt parameter to control output style and the correct spelling of names, brands, and terms. Explore how GPT can generate fictitious prompts for Whisper.

25 min

Speech-to-Text Methods: Batch, Streaming, and Realtime

Compare three OpenAI API transcription methods: blocking file upload, streaming (stream=True), and Realtime WebSocket. Understand the latency, complexity, and constraint trade-offs of each approach.

25 min

Module 3

Realtime API and Vision

Summarizing long voice conversations via the Realtime API and tagging images with GPT-4V.

Realtime API: Context Summarization for Voice Bots

Build a voice bot on the Realtime API with automatic summarization of old conversation turns: when the token window fills up, older messages are compressed into one and deleted from the server.

25 min

GPT-4V: Image Tagging and Captioning

Use GPT-4o-mini vision to generate tags and descriptions for e-commerce products. Deduplicate tags via embedding cosine similarity and turn descriptions into concise captions using few-shot prompting.

25 min