feat: add custom headers and base_url env expansion for providers

[alindsilva]

Summary

Add support for custom HTTP headers on provider and model configurations, enabling use cases like API gateways (e.g., Cloudflare AI Gateway) that require additional authentication or routing headers.

Changes

1. Custom headers in config (ee35e1a9)

• Add headers field (map[string]string) to both ProviderConfig and ModelConfig in the config schema
• Wire headers from config into ProviderOpts["headers"] in applyProviderDefaults()
• Model-level headers take precedence over provider-level headers
• Update JSON schema (agent-schema.json)
• Add tests for headers wiring logic

2. Forward Start/Stop to inner toolsets (19147d98)

• Fix "toolset not started" errors in multi-agent configurations
• Add Start()/Stop() forwarding to FilteredToolset, InstructionsToolset, and ToonToolset wrappers
• Add tests for all three wrapper types

3. Normalize anyOf schemas + API error logging (14eb42df)

• Add normalizeUnionTypes() to collapse single-element anyOf/oneOf arrays (required by some providers)
• Wire into ConvertParametersToSchema pipeline
• Add debug logging of API error response bodies in streaming handler
• Add schema normalization tests

4. Headers + base_url env expansion on all providers (11167d78)

• OpenAI: Read custom headers from ProviderOpts, expand ${VAR} env vars, auth middleware for custom providers
• Gemini: Custom headers via genai.HTTPOptions, base_url ${VAR} expansion
• Anthropic: Custom headers via option.WithHeader(), base_url ${VAR} expansion
• Gather env vars from header values and base_url in GatherEnvVarsForModels() so tests auto-set dummy values
• Fix old module path reference in test file
• Add example config (examples/custom_provider.yaml)

Example usage

providers:
  cloudflare_gateway:
    api_type: openai_chatcompletions
    base_url: https://gateway.ai.cloudflare.com/v1/acct/gw/compat
    token_key: GOOGLE_API_KEY
    headers:
      cf-aig-authorization: Bearer ${CLOUDFLARE_AI_GATEWAY_TOKEN}

models:
  gemini_via_cloudflare:
    provider: cloudflare_gateway
    model: google-ai-studio/gemini-3-flash-preview

Testing

• All existing tests pass
• New tests for: header wiring, schema normalization, Start/Stop forwarding
• TestLoadExamples passes with the new custom_provider.yaml example

[Copilot]

Pull request overview

This PR expands provider/model configurability and robustness across the runtime by adding configurable HTTP headers, environment-variable expansion for certain endpoints, schema normalization for stricter gateways, and lifecycle forwarding for wrapped toolsets.

Changes:

• Add headers fields to provider/model config (plus JSON schema + example) and wire them through provider defaults into per-provider HTTP client initialization (with env expansion in header values).
• Fix multi-agent “toolset not started” issues by forwarding Start()/Stop() through toolset wrapper types and adding tests.
• Normalize union/anyOf tool schemas for gateway compatibility and add additional API error detail logging for streaming failures.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/teamloader/toon.go	Forwards Start/Stop to inner toolset when supported.
pkg/teamloader/toon_test.go	Adds tests validating Start/Stop forwarding behavior for Toon wrapper.
pkg/teamloader/instructions.go	Forwards Start/Stop to inner toolset for instructions wrapper.
pkg/teamloader/instructions_test.go	Adds tests for Start/Stop forwarding behavior in instructions wrapper.
pkg/teamloader/filter.go	Forwards Start/Stop to inner toolset for filtered toolsets.
pkg/teamloader/filter_test.go	Adds tests for Start/Stop forwarding and startable integration chaining.
pkg/runtime/streaming.go	Adds debug logging for OpenAI SDK API error response dumps in streaming loop.
pkg/model/provider/provider.go	Wires provider/model headers into ProviderOpts["headers"] with model-level precedence.
pkg/model/provider/custom_headers_test.go	Adds tests covering provider/model header wiring into ProviderOpts.
pkg/model/provider/openai/client.go	Adds support for custom headers (with env expansion) and changes auth handling for custom providers without token_key.
pkg/model/provider/openai/api_type_test.go	Updates expectation to reflect stripping Authorization when no token_key is set.
pkg/model/provider/gemini/client.go	Adds base_url env expansion and custom headers via genai.HTTPOptions.
pkg/model/provider/anthropic/client.go	Adds base_url env expansion and custom headers via request options.
pkg/model/provider/openai/schema.go	Adds normalizeUnionTypes() into schema conversion pipeline.
pkg/model/provider/openai/schema_test.go	Adds tests for schema normalization of anyOf optional patterns.
pkg/model/provider/schema_test.go	Updates expected schema output to match normalized union handling.
pkg/config/latest/types.go	Adds headers to provider/model config types.
pkg/config/v3/types.go	Adds headers to provider/model config types (older version).
pkg/config/gather.go	Gathers env vars referenced in headers (and provider base_url) for missing-env checks.
agent-schema.json	Extends JSON schema with headers fields for provider/model.
examples/custom_provider.yaml	Updates example to demonstrate Cloudflare gateway headers usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[alindsilva]

Bug fix: Headers dropped during config upgrade chain

The original PR added Headers fields to ModelConfig and ProviderConfig in v3 and latest config types, but missed the intermediate versions (v4, v5, v6) that sit in the upgrade chain.

Since configs are migrated through the chain v3 → v4 → v5 → v6 → latest via CloneThroughJSON (JSON serialize → deserialize), the missing Headers field in v4 caused all custom headers to be silently dropped at the v3→v4 boundary. By the time the config reached the provider clients, ProviderOpts["headers"] was empty — so gateway auth headers like cf-aig-authorization were never sent, resulting in 401 Unauthorized errors from Cloudflare AI Gateway.

Fix (8d15048): Added Headers map[string]string to both ModelConfig and ProviderConfig in v4, v5, and v6 config types. This same commit also addresses all 6 Copilot review comments (see inline replies).