feat: Support AG-UI multimodal content parts (image, audio, video, document, binary)

[mikehole]

Summary

Upstream currently drops array-form UserMessage.content silently — only string content is forwarded, and any message with an image/audio/video/document/binary attachment ends up in the 400 "empty prompt" path. This PR adds canonical support for every AG-UI multimodal content-part type and wires extracted attachments into the OpenClaw agent runner's MediaPath* contract (same shape the msteams channel uses).

• Canonical AG-UI 0.0.52 types — image, audio, video, document all share {type, source:{type:"data"|"url", value, mimeType?}, metadata?} per @ag-ui/core's ImageInputContentSchema / AudioInputContentSchema / VideoInputContentSchema / DocumentInputContentSchema.
• Legacy 0.0.43 binary — still supported; accepts inline data or a data: URI on url.
• Temp files — written to os.tmpdir() under clawg-ui-<uuid>.<ext> and unlinked in a finally block regardless of dispatch success or failure.
• Categorized text marker — when a user message has no text part, a terse [user attached: 2 images, 1 document] is appended so the downstream prompt is non-empty and the LLM has a signal.
• 25 MB body cap (up from 1 MB) to accommodate multi-part inline base64.
• http(s) URLs rejected with 400 — SSRF / redirect handling / mid-stream size enforcement deserve a dedicated design, deferred to a follow-up. data: URIs are accepted.

Does not bump @ag-ui/core — runtime casts + small type-guards cover the 0.0.52 shapes without a dep bump. The bump is already being proposed in #26 and this PR stays intentionally independent.

Why this shape for the channel→runner contract

The existing channel comment (same pattern as msteams) is the strongest precedent for MediaPath/MediaUrl/MediaType (+ *s arrays). I've verified end-to-end only for image/*. Audio/video/document follow the same injection path with their native mimeType — if the current OpenClaw agent runner gates on image mimetypes and drops others, happy to narrow this PR's scope to images-only, add a runner-side fix in OpenClaw, or wait for your guidance. Either way the AG-UI-side plumbing is the same.

Non-overlap with open PRs

• Add reasoning and step reporting event surfacing #26 feature/reasoning — modifies dispatcher callbacks (outbound REASONING/STEP events). Zero overlap with inbound content-part extraction.
• fix: resolve gateway secret from OpenClaw credentials store #25 gateway-secret fix — auth path. Zero overlap.

Test plan

17 new cases in src/http-handler.test.ts under describe("Multimodal content parts", …):

• Parameterized happy-path: each of image / audio / video / document with inline data source → asserts MediaPath / MediaType / MediaUrl
• binary with inline data (image mimeType and non-image text/csv) → MediaPath + matching extension
• binary with data: URI in url field → parsed and saved
• Multiple mixed attachments → MediaPaths / MediaUrls / MediaTypes arrays in input order; scalar fields point to first attachment
• Body text carries categorized marker ([user attached: 2 images, 1 document])
• Empty-text + single attachment → body non-empty via marker
• http(s) URL in source.value or binary.url → 400 invalid_request_error
• URL rejection does not call resolveAgentRoute / dispatchReplyFromConfig
• Attachment parts on non-user roles → ignored (no MediaPath in ctx)
• Malformed parts (missing source / mimeType) → skipped, valid ones still processed
• Temp files unlinked after successful dispatch
• Temp files unlinked when the dispatcher throws

All 87 existing tests still pass (npm test).

Follow-ups (explicitly out of scope)

• Remote http(s) URL attachment fetching — deferred pending SSRF / redirect handling / mid-stream size enforcement design
• @ag-ui/core bump to 0.0.52 — coming via Add reasoning and step reporting event surfacing #26 or a dedicated PR

🤖 Generated with Claude Code

[contextablemark]

Interesting suggestion... thanks for the submission! I was thinking about this myself the other day and will try to take a look later today.

[contextablemark]

Thanks for this — I had some time to dig into the upstream openclaw/openclaw repo to verify the
contract and the channel-side plumbing here is correct. The MediaPath /
MediaUrl / MediaType (+ *s array) shape matches AgentMediaPayload in
src/plugin-sdk/agent-media-payload.ts exactly, and the runner consumes those
fields via src/agents/sandbox-media-paths.ts and
src/auto-reply/reply/stage-sandbox-media.ts. So audio/video/document should
flow end-to-end as far as this channel is concerned (modulo the configured
agent/model actually supporting the modality, which is the right caveat to
keep in the README).

Three small changes before merge:

1. Drop MediaFilename / MediaFilenames

AgentMediaPayload is:

type AgentMediaPayload = {
  MediaPath?: string;
  MediaType?: string;
  MediaUrl?: string;
  MediaPaths?: string[];
  MediaUrls?: string[];
  MediaTypes?: string[];
};

There's no MediaFilename(s) field - maybe this was something that previously existed and has since been removed - so the runner will silently ignore it.
Please remove the assignments in src/http-handler.ts (and the
MediaFilename assertion in src/http-handler.test.ts).

2. Use the SDK's buildAgentMediaPayload helper

The plugin SDK already exports buildAgentMediaPayload(items: { path, contentType }[])
which produces exactly the object you're constructing by hand. Swapping to it
keeps us aligned if the SDK ever adds fields (e.g. agent-scoped local roots,
transcripts):

import { buildAgentMediaPayload } from "openclaw/plugin-sdk";

const mediaPayload = extractedAttachments.length
  ? buildAgentMediaPayload(
      extractedAttachments.map((a) => ({ path: a.path, contentType: a.mimeType })),
    )
  : {};

3. Fix the msteams reference in the PR description

The PR body says "same shape the msteams channel uses" but extensions/msteams
doesn't appear in any of the upstream MediaPath hits (another possible case where something may have been removed upstream). The actual reference
implementations are extensions/whatsapp, qqbot, telegram, discord,
signal, googlechat, zalo. Worth correcting so future readers grep the
right place — e.g. "same shape the whatsapp/qqbot channels use, per the
AgentMediaPayload SDK type".

Otherwise this is a clean PR — good defensive design on the URL-rejection +
cleanup paths, and the test coverage at the channel boundary is thorough.
Happy to approve once the above are addressed.