LLM dispatch

Everything LLM-related lives under anton/core/llm/:

File	Role
provider.py	LLMProvider ABC + the wire types: LLMResponse, ToolCall, Usage, and the StreamEvent union (StreamTextDelta, StreamToolUseStart/Delta/End, StreamComplete, StreamTaskProgress, StreamToolResult, StreamContextCompacted)
anthropic.py	AnthropicProvider — direct Anthropic API
openai.py	OpenAIProvider — OpenAI, Azure-style, and any openai-compatible endpoint; carries a flavor (OpenAI / mdb.ai passthrough / generic compatible) that decides native web-tool routing
client.py	LLMClient — the facade the rest of the codebase calls
structured.py	Shared helpers for forced-tool-call structured output
prompt_builder.py	ChatSystemPromptBuilder — system prompt assembly
prompts.py	The base prompt templates
tracing.py	Per-turn TraceContext for outbound telemetry headers

Planning vs coding providers

Anton runs two provider/model pairs, configured independently in settings:

• planning — the user-facing turn loop. Smart, slower, more expensive.
• coding — fast/cheap structured work: scratchpad code help, memory compaction, identity extraction, consolidation, cerebellum post-mortems, history summarization.

LLMClient.from_settings() builds both from AntonSettings (planning_provider / planning_model / coding_provider / coding_model). Provider strings are anthropic, openai, or openai-compatible. For the compatible case, a flavor resolver distinguishes mdb.ai passthrough (when the base URL matches the configured Minds URL) from a generic third-party endpoint — that decides whether web_search/web_fetch execute server-side or fall back to handler-dispatched tools (see Tool system).

The artifact builder's own model

generate_artifact can build on a model of its own. With ANTON_ARTIFACT_MODEL set in the process environment, and optionally ANTON_ARTIFACT_REASONING_EFFORT, every planning, coding and structured call the pipeline makes asks for that model and effort, on the role's own provider and so its endpoint (model_route and ModelRoute in client.py, entered inside the pipeline's task). The effort goes out as set, and an empty effort sends none, not the provider's own. The agent's and router's calls in the same turn keep their own models. The builder's usage is reported under its model, its calls don't change the served model behind the agent's identity line, a screenshot refusal by the builder's model turns off screenshots for that model alone, and the step trace's llm_call events name the model each call asked for.

LLMClient surface

Method	Provider	Returns
plan(...) / plan_stream(...)	planning	LLMResponse / StreamEvent iterator — the main turn loop
code(...)	coding	LLMResponse — free-text completions on the cheap model
generate_object(Schema, ...)	planning	A validated Pydantic instance
generate_object_code(Schema, ...)	coding	Same, on the cheap model

Structured output: generate_object

Anton has a single primitive for getting structured data out of the LLM. How it works:

1. The Pydantic model is converted to a JSON schema via model_json_schema().
2. A synthetic tool is built whose input_schema is that schema.
3. The provider is called with tool_choice={"type": "tool", "name": tool_name} — this forces the LLM to call the tool rather than returning text.
4. Some models refuse a forced choice with a 400 that names tool_choice: Claude Opus 5.5, Sonnet 5.5 and Fable 5.1, and an older Claude with thinking on. send_structured then sends the call once more with no tool_choice and a line asking for the tool first, and remembers the refusal for that endpoint and model while the process runs, so later calls there skip the forced choice. An unforced reply with no tool call gets one more try, so a call sends at most three requests.
5. The tool-call input is validated through model_validate() and returned as a typed instance (or list[Model] for list annotations).

Why it beats asking for JSON in text

Old pattern (text JSON)	New pattern (generate_object)
"Return ONLY valid JSON, no commentary, no markdown fences"	Forced tool_choice — the LLM cannot return text
Manual json.loads() with try/except	Pydantic model_validate() with structural validation
Strip markdown fences with regex	Never needed — there's no text response to strip
Defensive isinstance checks	Pydantic catches type errors at the schema layer
Field-by-field .get(key, default) extraction	Typed attribute access on the validated instance

The shared helper: structured.py

Schema derivation and validation live in exactly one place — two pure functions shared by LLMClient.generate_object (main process, async) and _ScratchpadLLM.generate_object (subprocess bridge, sync — see Scratchpad runtime):

Function	Purpose
build_structured_tool(schema_class)	Pydantic model (or list[Model]) → (tool_dict, validator_class, is_list). List inputs are wrapped in a synthetic _ArrayWrapper with an items field, because many providers refuse top-level arrays in tool input schemas
unwrap_structured_response(tool_call_input, validator_class, is_list)	Validate via Pydantic and unwrap the list wrapper. Raises pydantic.ValidationError on schema drift

Pydantic is imported lazily so the module is importable without it.

Adding a new extraction call site is mechanical: define a Pydantic model with Field(description=...) on each field (the descriptions double as the LLM's instructions — no separate prompt explains the schema), call await session._llm.generate_object(MySchema, ...), wrap in try/except for graceful degradation.

Where generate_object is used

Module	Schema	Provider	Purpose
connect_collector.py::extract_variables	_ExtractionResult	planning	Parse free-form credential input into structured fields
commands/skills.py::handle_skill_save	_SkillDraft	planning	Draft a skill from recent scratchpad work
commands/datasource.py::handle_add_custom_datasource	_CustomDatasourceSpec	planning	Identify a custom datasource's auth fields
cortex.py::plan_compaction	_CompactionResult	coding	Memory deduplication during synaptic homeostasis
cortex.py::maybe_update_identity	_IdentityFacts	coding	Default-mode identity extraction every 5 turns
consolidator.py::replay_and_extract	_ConsolidatedLessons	coding	Sleep-replay extraction of lessons
cerebellum.py::_run_diff	_DiffPassResult	coding	Post-mortem error learning from cell failures

The planning/coding split preserves each call site's original intent: anything that previously used _llm.code() now uses generate_object_code, anything that used _llm.plan() uses generate_object.

Prompt assembly: prompt_builder.py

ChatSystemPromptBuilder.build(...) concatenates, in order:

1. prefix (from SystemPromptContext)
2. The base CHAT_SYSTEM_PROMPT (runtime identity, visualization rules)
3. Tool prompts — optional ToolDef.prompt fragments
4. Procedural memory section — one line per skill from SkillStore.list_summaries(), telling the LLM to call recall_skill(label)
5. Project context — anton.md
6. Self-awareness context (legacy) and datasource context
7. suffix
8. Memory context — cortex.build_memory_context() output, the operator's read-only layer first when the host passes one
9. Workspace discovery

Items 5 to 9 follow a session marker. Everything above memory stays byte-stable across turns, so the provider's prompt cache holds it; memory and workspace discovery change every turn, so they ride the tail.

Trace headers

tracing.py defines a TraceContext (session_id, turn_id, harness) held in a ContextVar. ChatSession.turn_stream installs it for the duration of a turn, so nested calls — structured output, the cerebellum's diff, scratchpad coding calls in the same task — inherit the same trace without threading kwargs.

The OpenAI provider reads it when talking to MindsHub and attaches Langfuse-Session-Id, Langfuse-Tags, and Langfuse-Metadata headers. Against any other openai-compatible endpoint the headers are opt-in via ANTON_LANGFUSE_HEADERS=1. Direct Anthropic and raw OpenAI ignore the context. See Trace headers.