Skip to content

README.md

Latest commit

 

History

History
289 lines (216 loc) · 11.7 KB

README.md

File metadata and controls

289 lines (216 loc) · 11.7 KB
Raw

Shipyard Agent

Autonomous coding-agent server that turns a natural-language task into a traced LangGraph run: plan, coordinate isolated workers, verify with deterministic gates, review, retry or recover, then report evidence.

What This Proves

  • Agent orchestration: a LangGraph StateGraph routes gate -> plan -> coordinate/execute -> verify -> review -> error_recovery -> report, with worker coordination as the default path and single-agent execution as recovery fallback.
  • Production AI runtime: Express API, WebSocket live feed, optional Postgres persistence, auth/rate limits, GitHub connector and webhook/invoke/retry surfaces, LangSmith trace resolution, and run-level telemetry.
  • Reliability discipline: deterministic verification, file-overlay rollback on retry, scoped tool boundaries, failure catalog, benchmark renderer, load tests, and a done gate that enforces typecheck plus a minimum test baseline.

Shipyard Agent orchestration workflow

Watch the 8-second workflow video

Shipyard agent workflow

Demo

Reproducible local demo: docs/demo.md

Expected run shape:

{
  "runId": "9b5c...",
  "phase": "done",
  "steps": [
    { "index": 0, "description": "Inspect target files", "status": "done" },
    { "index": 1, "description": "Apply focused edit", "status": "done" },
    { "index": 2, "description": "Add regression test", "status": "done" }
  ],
  "verificationResult": { "passed": true, "newErrorCount": 0 },
  "traceUrl": "https://smith.langchain.com/..."
}

Live surfaces:

  • Dashboard: /dashboard
  • Runs: /runs
  • Benchmarks: /benchmarks
  • Health: /api/health
  • API reference: docs/API-REFERENCE.md

Quickstart

pnpm install
cp .env.example .env
pnpm dev

Server starts on http://localhost:4200.

curl http://localhost:4200/api/health

Run the deterministic checks:

pnpm type-check
pnpm test

Production-style local start:

pnpm build
pnpm start

Optional features:

  • OPENAI_API_KEY is required for real agent runs with the default provider routing.
  • ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN enables Anthropic model selections.
  • SHIPYARD_DB_URL enables Postgres-backed run persistence; without it, Shipyard uses memory.
  • LANGCHAIN_TRACING_V2=true plus LANGCHAIN_API_KEY enables LangSmith tracing.
  • SHIPYARD_ENABLE_WEB_SEARCH=true plus BRAVE_SEARCH_API_KEY enables the guarded web-search tool.

Architecture

flowchart LR
  User["User, dashboard, API, GitHub webhook"] --> API["Express API + WebSocket"]
  API --> Gate["gate node"]
  Gate --> Plan["planner"]
  Plan --> Coord["coordinator"]
  Coord --> Workers["isolated worker agents"]
  Plan --> Execute["single-agent fallback"]
  Workers --> Verify["verify: typecheck + tests"]
  Execute --> Verify
  Verify --> Review["reviewer"]
  Review -->|done| Report["report"]
  Review -->|continue| Execute
  Review -->|retry| Plan
  Review -->|escalate| Recovery["error_recovery"]
  Recovery -->|retry planning| Plan
  Recovery -->|retry execution| Execute
  Recovery -->|fatal| Report
  Workers --> Tools["tool registry"]
  Execute --> Tools
  Tools --> Files["target repo files"]
  Tools --> Shell["guarded bash"]
  Tools --> Search["grep, glob, ls, web_search"]
  API --> Store["Postgres or in-memory store"]
  API --> Metrics["metrics, audit, dead letter"]
  Report --> Trace["LangSmith trace URL"]
Loading

Request lifecycle:

  1. Client submits an instruction through POST /api/run, WebSocket submit, invoke route, or GitHub connector flow.
  2. gate classifies chat vs code work and chooses plan, coordinate, execute, or end.
  3. plan decomposes the instruction into vertical steps and target files.
  4. coordinate dispatches worker agents for planned steps, isolates worker state, merges edits, and falls back when file ownership conflicts.
  5. Tools can read, search, edit, write, run guarded bash, inject context, ask the user, spawn agents, and open PRs through explicit adapters.
  6. verify runs deterministic checks and records baseline vs newly introduced errors.
  7. review decides continue, done, retry, or escalate; retry paths use feedback and file-overlay rollback.
  8. report returns final state, cost estimate, next actions, and LangSmith trace URL when tracing is enabled.

Agent Design

Core state lives in src/graph/state.ts and tracks the instruction, plan steps, file edits, tool-call history, verification result, review decision, retry count, trace URL, token usage, worker results, and loop diagnostics.

Primary loop:

Stage Code Responsibility
Gate src/graph/nodes/gate.ts Route chat, code, continuation, and direct execution paths.
Plan src/graph/nodes/plan.ts Inspect repo context and create executable steps.
Coordinate src/graph/nodes/coordinate.ts, src/multi-agent/* Decompose step work, run workers, merge edits, handle conflicts.
Execute src/graph/nodes/execute.ts Single-agent tool loop used as fallback or sequential mode.
Verify src/graph/nodes/verify.ts Run typecheck/tests and compute new vs pre-existing errors.
Review src/graph/nodes/review.ts Deterministic guards plus model review for continue/done/retry/escalate.
Recovery src/graph/nodes/error-recovery.ts Retry planning/execution or stop with a fatal report.
Report src/graph/nodes/report.ts Summarize final state, cost, trace, and next actions.

Tool boundaries:

  • File tools: read_file, edit_file, write_file, revert_changes.
  • Search tools: grep, glob, ls, optional web_search.
  • Runtime tools: bash, inject_context, ask_user, spawn_agent, commit_and_open_pr.
  • Edit safety: edit_file uses a four-tier cascade: exact match, whitespace-normalized match, Levenshtein fuzzy match, then full rewrite as logged last resort.
  • Bash safety: dangerous commands are blocked and covered by tests.

Failure recovery:

  • Retries carry structured review feedback back to planning or execution.
  • File overlays allow rollback when review chooses retry.
  • Watchdogs detect no-edit stalls, repeated tool loops, ambiguous edits, provider errors, auth errors, and max tool rounds.
  • Coordinator conflict handling falls back to safer execution when parallel workers touch shared files.
  • Dead-letter and retry APIs preserve failed invoke/webhook events for replay.

API Surface

Core run APIs:

curl -X POST http://localhost:4200/api/run \
  -H "Content-Type: application/json" \
  -d '{"instruction":"Add a regression test for the parser edge case"}'

curl http://localhost:4200/api/runs/<run-id>

curl -X POST http://localhost:4200/api/runs/<run-id>/followup \
  -H "Content-Type: application/json" \
  -d '{"instruction":"Now explain the tradeoff"}'

Additional surfaces are documented in docs/API-REFERENCE.md: invoke batch/retry, GitHub webhooks, dead-letter replay, checkpoints, metrics, readiness, benchmark snapshots, model key settings, and GitHub App repo connection.

Error behavior is structured for automation:

{
  "error": "Human-readable message",
  "code": "MACHINE_CODE",
  "details": { "field": "context" }
}

Observability

  • LangSmith integration is in src/runtime/langsmith.ts; it supports LANGCHAIN_* and LANGSMITH_* env vars.
  • SHIPYARD_TRACE_PUBLIC=true creates shareable public trace links for demos and benchmarks.
  • Tool calls, node decisions, parser decisions, token usage, cost, trace URLs, audit events, and live WebSocket events are tracked.
  • Dashboard live feed documents file-edit and tool-activity streaming in docs/DASHBOARD-LIVE-FEED.md.
  • Metrics and audit behavior are covered in docs/SECURITY.md and docs/ops-metrics.md.

Evals And Benchmarks

Run tests:

pnpm test

Run the repo done gate:

pnpm done:check

Benchmark a seeded instruction:

./scripts/bench.sh 01-strict-typescript
pnpm exec tsx scripts/render-benchmarks.ts
pnpm exec tsx scripts/render-issues.ts

Evidence:

Current benchmark docs include a verified 7/7 step rebuild sequence, LangSmith trace links, token/edit totals, recent swarm failures, and known open failure classes.

Deployment

Shipyard is a Node service:

pnpm build
pnpm start

Current deployment notes in this repo name Hostinger as the live host and expose:

  • Dashboard: https://agent.ship.187.77.7.226.sslip.io/dashboard
  • Base URL: https://agent.ship.187.77.7.226.sslip.io

No Dockerfile or Compose file is currently present. Deployment proof is therefore the Node build/start path, env contract in .env.example, health/readiness routes, optional Postgres persistence, and the live Hostinger endpoint.

Security And Privacy

Read docs/SECURITY.md before exposing beyond localhost.

Implemented surfaces include:

  • Global API bearer token through SHIPYARD_API_KEY.
  • Invoke/retry bearer or X-Shipyard-Invoke-Token.
  • GitHub webhook HMAC via SHIPYARD_GITHUB_WEBHOOK_SECRET.
  • Header sanitization for dead-letter persistence.
  • Bot-loop prevention and webhook sender allowlists.
  • Dashboard XSS hardening on user-controlled preview fields.
  • Rate limits on run/followup/write endpoints.

Remote multi-user deployments still need full same-origin CSRF tokens for every state-changing route.

Evidence Index

Tradeoffs

  • Worker orchestration improves context isolation but can introduce shared-file conflicts; Shipyard detects overlap and falls back to safer sequential execution.
  • edit_file optimizes for surgical changes, but tier-4 full rewrites remain a last-resort escape hatch and are logged as degraded edits.
  • LangSmith public trace links are excellent demo evidence but should be disabled with SHIPYARD_TRACE_PUBLIC=false for sensitive work.
  • Docker packaging is not yet represented in-repo; production deploy evidence currently depends on Node build/start plus the external Hostinger host.

Repo Map

src/graph/          LangGraph builder, state, routing, and node implementations
src/multi-agent/    Worker orchestration, isolated worktrees, merge/materialization
src/tools/          Tool registry and adapters
src/runtime/        Loop, persistence, checkpoints, LangSmith, shutdown
src/server/         Express routes, dashboard, WebSocket, GitHub, invoke/retry
scripts/            Benchmarks, load tests, migrations, done gate, renderers
test/               Vitest coverage for graph, runtime, server, tools, scripts
docs/               API, security, benchmarks, reliability, plans, demo assets

License

No license file is currently present in this repository.