第 6 章：多模态
出处
来源：multimodal/ · 状态：已读
一句话
图不是「贴一张让模型看」，是另一种要钉形状的输入：怎么传、什么分辨率、提示写在图上还是字上、细节不够就给放大镜。 幻灯和扫描件尤其不能靠传统 OCR 交差。
核心概念
传图：URL 就能进 Messages API
getting_started_with_vision.ipynb 是最小路径。后面所有技巧都建立在「图已经是一条 content block」。
视觉也吃传统 prompt 工程
best_practices_for_vision.ipynb：数狗会数错，加角色和约束就对了。还可以 把提示画在图上（箭头、高亮、图内问题）。few-shot 用图当例子（读时速表单位、认裤型）。超长收据切成多图一次喂。视觉 few-shot 不是总有效，但值得试。
图表和幻灯：算术和颜色是坑
reading_charts_graphs_powerpoints.ipynb：图表优先走 PDF + vision（当时 pdfs-2024-09-25 beta，模型限制以 notebook 为准）。两个坑：算术偶发错 → 给计算器工具；靠颜色分组的图 → 先让模型报 HEX 再读数。复杂图可先「描述每一个数据点」（视觉版 CoT）。
幻灯：PyPDF 抽文本会丢掉真正有信息的图。整本 PDF 直接喂有页数上限（当时 100 页），且多模态 PDF 进 RAG embedding 会惹麻烦。替代：逐页叙述，当前页 + 已有叙述，得到可检索的文本稿。
OCR 是「按你的形状抽」，不是认字
how_to_transcribe_text.ipynb：优势是指定抽什么（只要答案里的代码、手写处方、表格单）。再往上是文档 QA，以及非结构 → JSON（接回第 1 章 schema）。
Zoom：固定分辨率看不清时，给 lean-in
crop_tool.ipynb 是本章最硬的工程。Claude 按约 28×28 patch 看图，细标签、紧挨的线在这个分辨率下 prompt 救不回来。zoom 工具：模型输出像素框，你从 原图全分辨率 crop 并放大再回传。
必须先按文档的 resized_size() 预缩放，模型吐的坐标才和你手里的图 1:1。Chartography（100 题）：Fable 5 29% → 73%（+44）；Sonnet 5 13% → 44%（+31）。演示题 Widget-A/E：无工具 40 次里 36 次错，有工具 40/40。代价：每 zoom 重送对话 + 放大作物，Fable 5 约 $0.08 → $0.95 / 题。长 thinking 轮要注意 stop-reason，避免静默截断。
同一模式适用密文档、UI 截图、原理图、扫描表单、标签照片。视觉叠工具见 03-工具调用 的 vision_with_tools。
语音是进出适配器
third_party/Deepgram/prerecorded_audio.ipynb、ElevenLabs/low_latency_stt_claude_tts.ipynb：录音 → 文本 → Claude → 语音。能力边界在 STT/TTS 厂商，Claude 仍吃文本。细节见 12-集成与第三方。
对照 recipe（默认折叠；机制片段摘自官方 notebook，全文在 GitHub）：
multimodal/getting_started_with_vision.ipynb — 图怎么进 Messages API
python
import base64
from anthropic import Anthropic
client = Anthropic()
MODEL_NAME = "claude-opus-4-8"
with open("../images/sunset.jpeg", "rb") as image_file:
binary_data = image_file.read()
base_64_encoded_data = base64.b64encode(binary_data)
base64_string = base_64_encoded_data.decode("utf-8")
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": "image/jpeg", "data": base64_string},
},
{"type": "text", "text": "Write a sonnet based on this image."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
import httpx
IMAGE_DATA = base64.b64encode(httpx.get(IMAGE_URL).content).decode("utf-8")
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": "image/jpeg", "data": IMAGE_DATA},
},
{"type": "text", "text": "Describe this image in two sentences."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)multimodal/best_practices_for_vision.ipynb — 角色、图内提示、few-shot、切图
python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/70.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "You are going 70 miles per hour."}],
},
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/100.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "You are going 100 miles per hour."}],
},
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/140.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
]
response = client.messages.create(
model=MODEL_NAME, max_tokens=2048, messages=message_list, temperature=0
)
print(response.content[0].text)multimodal/reading_charts_graphs_powerpoints.ipynb — 图表算术 / 颜色；幻灯改叙述再检索
python
# Make a useful helper function.
def get_completion(messages):
response = client.messages.create(
model=MODEL_NAME, max_tokens=8192, temperature=0, messages=messages
)
return response.content[0].textmultimodal/how_to_transcribe_text.ipynb — 按形状抽，不按「认全页」
python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/transcribe/stack_overflow.png"),
},
},
{"type": "text", "text": "Transcribe the code in the answer. Only output the code."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/transcribe/vehicle_form.jpg"),
},
},
{"type": "text", "text": "Transcribe this form exactly."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/transcribe/page.jpeg"),
},
},
{"type": "text", "text": "Which is the most critical issue for live rep support?"},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)multimodal/crop_tool.ipynb — 给模型一把放大镜
python
from anthropic import beta_tool
from anthropic.types.beta import BetaContentBlockParam, BetaMessageParam
def ask_with_zoom_tool(images, question: str, verbose: bool = True, model: str = MODEL):
"""Ask Claude a question about one or more images, with the zoom tool available."""
if not isinstance(images, list):
images = [images]
originals = images
views = [prepare_image(image) for image in originals]
@beta_tool
def zoom(x1: int, y1: int, x2: int, y2: int, image_index: int = 0) -> list:
"""Zoom into a rectangular region of an image to see it magnified.
Use this whenever text, numbers, lines, or details are too small to read
confidently in the full view. The region is cropped from the image's
full-resolution original and scaled up to fill the image budget, so
small elements become much easier to read. Coordinates are absolute
pixels in the image as you see it, origin at the top-left corner.
Zoom again on the result for even finer detail.
Args:
x1: Left edge of the region in pixels (origin is the image's top-left corner).
y1: Top edge of the region in pixels.
x2: Right edge of the region in pixels (must be greater than x1).
y2: Bottom edge of the region in pixels (must be greater than y1).
image_index: Which image to zoom into (0 for the first image, 1 for the second).
"""
if not 0 <= image_index < len(originals):
return [{"type": "text", "text": f"Error: image_index {image_index} is out of range"}]
return handle_zoom(originals[image_index], views[image_index].size, x1, y1, x2, y2)
content: list[BetaContentBlockParam] = []
for i, view in enumerate(views):
content.append({"type": "text", "text": f"Image {i} ({view.width}x{view.height} pixels):"})
content.append(
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": pil_to_base64(view),
},
}
)
content.append(
{
"type": "text",
"text": (
f"Answer the following question about the image{'s' if len(views) > 1 else ''} "
f"above. Coordinates are absolute pixels with the origin at the top-left corner. "
f"Use the zoom tool to examine any detail that is too small to read confidently."
f"\n\nThe question is: {question}"
),
}
)
messages: list[BetaMessageParam] = [{"role": "user", "content": content}]
runner = final = None # bound before the loop for the type checker; the body always runs
for _reentry in range(5): # pause_turn re-entries are bounded, like the tool loop itself
runner = client.beta.messages.tool_runner(
model=model,
max_tokens=8192,
tools=[zoom],
messages=messages,
max_iterations=20,
)
for message in runner:
called = [block for block in message.content if block.type == "tool_use"]
if verbose:
for block in called:
print(f"[Tool] {block.name}({block.input})")
if called:
# The runner caches this response and reuses it internally, so
# peeking at it here does not re-run the tool. It is the
# tool_result message the runner sends next; its blocks are the
# plain dicts handle_zoom() built, so read them as such.
peek = cast(Any, runner.generate_tool_call_response())
for tool_result in peek["content"] if peek else []:
# An errored tool call carries a plain string instead
# of blocks; only successful zooms have images to show.
if isinstance(tool_result["content"], list):
show_tool_result(tool_result["content"], show_text=False)
final = runner.until_done()
if final.stop_reason != "pause_turn":
break
# The server paused a long turn; the runner doesn't resume these itself,
# so replay the transcript so far and continue where it left off. The
# counter records how often that happened.
globals().setdefault("PAUSE_TURN_RESUMES", []).append(_reentry + 1)
snapshot = {}
runner.set_messages_params(lambda params, _out=snapshot: (_out.update(params), params)[1])
messages = [*snapshot["messages"], {"role": "assistant", "content": final.content}]
assert runner is not None and final is not None # the loop ran at least once
answer = "\n".join(block.text for block in final.content if block.type == "text")
if not answer.strip():
# The run ended without a text answer - usually the iteration budget ran
# out mid-zoom. Give the model one plain turn to answer from what it has
# (the same wrap-up the hand-written loop in the appendix uses). When the
# budget ran out, the runner has already appended the final assistant
# message and its tool results to the transcript, so we only append the
# wrap-up request itself (appending the final message again would
# duplicate its tool calls and the API would reject the transcript).
snapshot = {}
runner.set_messages_params(lambda params, _out=snapshot: (_out.update(params), params)[1])
transcript = list(snapshot["messages"])
if not (transcript and transcript[-1]["role"] == "user"):
transcript.append({"role": "assistant", "content": final.content})
transcript.append(
{
"role": "user",
"content": (
"You have used your zoom budget. Answer the original question now, "
"based on everything you have seen."
),
}
)
wrap_up = client.beta.messages.create(model=model, max_tokens=8192, messages=transcript)
final = wrap_up
answer = "\n".join(block.text for block in final.content if block.type == "text")
if verbose:
for block in final.content:
if block.type == "text":
print(f"[Assistant] {block.text}")
# Join every text block: long answers can arrive in several, and the
# answer is not always in the last one.
return answerthird_party/Deepgram/prerecorded_audio.ipynb — 录音进文本再交给 Claude
python
import json
import anthropic
transcription_file = "transcript.json"
# Function to get the transcript from the JSON file
def get_transcript(transcription_file):
with open(transcription_file) as file:
data = json.load(file)
result = data["results"]["channels"][0]["alternatives"][0]["transcript"]
return result
# Load the transcript from the JSON file
message_text = get_transcript(transcription_file)
# Initialize the Claude API client
client = anthropic.Anthropic(
# Defaults to os.environ.get("ANTHROPIC_API_KEY")
# Claude API key
api_key="🔑🔑🔑 Your API Key here! 🔑🔑🔑"
)
# Prepare the text for the API request
formatted_messages = [{"role": "user", "content": message_text}]
# Generate thoughtful, open-ended interview questions
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=1000,
temperature=0.5,
system="Your task is to generate a series of thoughtful, open-ended questions for an interview based on the given context. The questions should be designed to elicit insightful and detailed responses from the interviewee, allowing them to showcase their knowledge, experience, and critical thinking skills. Avoid yes/no questions or those with obvious answers. Instead, focus on questions that encourage reflection, self-assessment, and the sharing of specific examples or anecdotes.",
messages=formatted_messages,
)
# Print the generated questions
# Join the text of each TextBlock into a single string
content = "".join(block.text for block in response.content)
# Split the content by '\n\n'
parts = content.split("\n\n")
# Print each part with an additional line break
for part in parts:
print(part)
print("\n")third_party/ElevenLabs/low_latency_stt_claude_tts.ipynb — 低延迟语音进出
python
start_time = time.time()
message = anthropic_client.messages.create(
model="claude-haiku-4-5",
max_tokens=1000,
temperature=0,
messages=[{"role": "user", "content": transcription.text}],
)
end_time = time.time()
non_streaming_response_time = end_time - start_time
print(message.content[0].text)
print(f"\nResponse time: {non_streaming_response_time:.2f} seconds")关键洞察
多模态的失败经常被说成「模型看不懂」。cookbook 的诊断更具体：分辨率不够、颜色没先标定、算术没工具、幻灯当纯文本切了。zoom 把「看」收成工具环——和第 3 章同一逻辑：模型提出需要，你的工具改变它能看见的世界。
准确率涨幅很大，账单也是。视觉工具是产品决策，不是默认打开的滤镜。
个人思考
第 5 章用 Haiku 读财报表格图，已经是多模态工作流。本章补的是单模型看不清时怎么办。Skills（下一章）会把「生成 xlsx/pptx/pdf」收成包；和「读这些格式」是对称的两面。