-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathtavily-search-known-source.html
More file actions
163 lines (154 loc) · 16.9 KB
/
Copy pathtavily-search-known-source.html
File metadata and controls
163 lines (154 loc) · 16.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Agent Royale Report</title>
<style>
:root {
color-scheme: light;
--ink:#15181e; --muted:#667085; --line:#d9dee8; --soft:#f4f7fb;
--panel:#ffffff; --good:#087f5b; --bad:#c92a2a; --warn:#b76e00; --blue:#2563eb;
}
* { box-sizing:border-box; }
body { margin:0; font-family:Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; color:var(--ink); background:linear-gradient(180deg,#f8fbff 0,#ffffff 220px); }
.wrap { max-width:1440px; margin:0 auto; padding:22px 24px 44px; }
header { display:grid; grid-template-columns:minmax(0,1fr) 170px; gap:18px; align-items:start; }
h1 { margin:0 0 6px; font-size:28px; line-height:1.08; letter-spacing:0; }
h2 { margin:20px 0 10px; font-size:18px; letter-spacing:0; }
p { margin:0; color:var(--muted); line-height:1.5; }
code { font-family:"SFMono-Regular", Consolas, monospace; font-size:12px; }
.subhead { max-width:880px; font-size:14px; }
.meta { display:flex; flex-wrap:wrap; gap:7px; margin-top:12px; }
.pill { border:1px solid var(--line); background:rgba(255,255,255,.88); border-radius:999px; padding:6px 9px; color:#344054; font-size:11px; font-weight:700; }
.verdict { border:1px solid var(--line); background:var(--panel); border-radius:8px; padding:13px; box-shadow:0 10px 28px rgba(15,23,42,.05); }
.score { font-size:34px; line-height:1; font-weight:850; color:var(--bad); }
.verdict-label { margin-top:6px; font-size:12px; font-weight:800; color:var(--ink); }
.grid { display:grid; grid-template-columns:repeat(5,minmax(0,1fr)); gap:10px; margin-top:16px; }
.card { border:1px solid var(--line); background:var(--panel); border-radius:8px; padding:12px; box-shadow:0 7px 20px rgba(15,23,42,.035); }
.num { font-size:24px; line-height:1; font-weight:850; }
.label { color:var(--muted); font-size:10px; text-transform:uppercase; letter-spacing:.08em; margin-top:6px; font-weight:800; }
.insight { margin-top:16px; border:1px solid #c7d7fe; background:#eff6ff; border-radius:8px; padding:14px 16px; display:grid; grid-template-columns:128px minmax(0,1fr); gap:14px; align-items:start; }
.insight strong { color:#1d4ed8; font-size:12px; text-transform:uppercase; letter-spacing:.08em; }
.method { display:grid; grid-template-columns:repeat(3,minmax(0,1fr)); gap:12px; margin-top:16px; }
.method-card { border:1px solid var(--line); background:#fff; border-radius:8px; padding:14px 16px; }
.method-card strong { display:block; font-size:12px; text-transform:uppercase; letter-spacing:.08em; color:#344054; margin-bottom:6px; }
.split { display:grid; grid-template-columns:.82fr 1.18fr; gap:22px; align-items:start; }
table { width:100%; border-collapse:separate; border-spacing:0; background:var(--panel); border:1px solid var(--line); border-radius:8px; overflow:hidden; box-shadow:0 8px 24px rgba(15,23,42,.04); }
th,td { text-align:left; padding:10px 11px; border-bottom:1px solid #e8ecf3; vertical-align:top; font-size:13px; }
th { font-size:10px; text-transform:uppercase; letter-spacing:.06em; color:var(--muted); background:#f7f9fc; font-weight:850; }
tr:last-child td { border-bottom:0; }
.pass,.fail { font-weight:850; }
.pass { color:var(--good); }
.fail { color:var(--bad); }
.muted { color:var(--muted); }
.answer { max-width:310px; white-space:pre-wrap; color:#344054; }
.question { color:var(--muted); font-size:12px; line-height:1.35; max-width:340px; margin-top:5px; }
.model { max-width:210px; overflow-wrap:anywhere; font-size:12px; color:#344054; }
.claim { font-weight:750; }
.truth { font-weight:750; color:#175cd3; }
.failure { color:var(--bad); font-weight:750; }
.source { color:#475467; font-size:12px; max-width:320px; overflow-wrap:anywhere; }
.citation-list { margin-top:6px; display:grid; gap:3px; }
.required { font-weight:800; color:#344054; }
.catch-list { display:grid; gap:10px; }
.catch { border:1px solid var(--line); border-left:4px solid var(--bad); border-radius:8px; background:#fff; padding:13px 14px; }
.catch-title { display:flex; justify-content:space-between; gap:14px; align-items:center; margin-bottom:8px; }
.catch-title code { color:var(--bad); font-weight:800; }
.catch-grid { display:grid; grid-template-columns:1fr 1fr; gap:10px; }
.mini-label { color:var(--muted); font-size:11px; text-transform:uppercase; letter-spacing:.06em; margin-bottom:3px; font-weight:800; }
.status-dot { display:inline-block; width:8px; height:8px; border-radius:999px; margin-right:8px; background:var(--muted); }
.status-dot.correct,.status-dot.source-supported-correct { background:var(--good); }
.status-dot.wrong-value,.status-dot.wrong-source,.status-dot.unsupported-citation,.status-dot.no-answer { background:var(--bad); }
footer { margin-top:24px; color:var(--muted); font-size:12px; }
@media(max-width:980px) { header,.split,.method { grid-template-columns:1fr; } .grid { grid-template-columns:1fr 1fr; } .insight { grid-template-columns:1fr; } th:nth-child(7),td:nth-child(7) { display:none; } }
</style>
</head>
<body>
<main class="wrap">
<header>
<div>
<h1>Agent Royale Eval Report</h1>
<p class="subhead">This run tested one agent target against 5 live-web tasks. The table below shows the stack, model, or adapter target called, question, answer returned, extracted claim, ground truth, and citations.</p>
<div class="meta">
<span class="pill">target: http://127.0.0.1:3013/api/agent</span>
<span class="pill">runs: 5</span>
<span class="pill">generated: 2026-06-06</span>
</div>
</div>
<div class="verdict">
<div class="score">20%</div>
<div class="verdict-label">High risk</div>
<p>Exact-value pass rate across 5 scoreable runs</p>
</div>
</header>
<section class="grid">
<div class="card"><div class="num">1</div><div class="label">Exact values</div></div>
<div class="card"><div class="num">1</div><div class="label">Source-supported</div></div>
<div class="card"><div class="num">4</div><div class="label">Issues</div></div>
<div class="card"><div class="num">3</div><div class="label">No answer</div></div>
<div class="card"><div class="num">0</div><div class="label">Oracle skips</div></div>
</section>
<h2>Run Details</h2>
<table>
<colgroup>
<col style="width:5%">
<col style="width:23%">
<col style="width:12%">
<col style="width:17%">
<col style="width:15%">
<col style="width:10%">
<col style="width:8%">
<col style="width:10%">
</colgroup>
<thead><tr><th>Status</th><th>Question</th><th>Stack / Target</th><th>Answer returned</th><th>Extracted claim</th><th>Ground truth</th><th>Outcome</th><th>Citations / source</th></tr></thead>
<tbody><tr><td><span class="fail">FAIL</span></td><td><code>stackfit_known_next_package_manager</code><div class="question">In the canary Next.js package.json file on GitHub, what exact packageManager value is listed?</div></td><td class="model">http://127.0.0.1:3013/api/agent</td><td class="answer">The packageManager value in the canary Next.js package.json is npm.</td><td class="claim">npm</td><td class="truth">pnpm@10.33.0<div class="muted" style="font-size:11px;margin-top:4px">Evidence: pnpm@10.33.0</div></td><td><div class="failure">wrong value</div><div class="muted" style="font-size:11px;margin-top:3px">Oracle: verified</div></td><td class="source"><div><span class="required">Required:</span> github.com/vercel/next.js/blob/canary/package.json</div><div class="citation-list"><span class="required">Cited:</span><div>github.com/vercel/next.js/blob/canary/package.json</div><div>github.com/vercel/next.js/issues/85587</div><div>github.com/vercel/next.js/blob/canary/packages/next/package.json</div><div class="muted">+2 more</div></div></td></tr>
<tr><td><span class="fail">FAIL</span></td><td><code>stackfit_known_playwright_release_tag</code><div class="question">On the microsoft/playwright GitHub Releases page, which latest release tag is listed?</div></td><td class="model">http://127.0.0.1:3013/api/agent</td><td class="answer">{"query": "Return the exact single value for this source-specific task. Do not abbreviate numbers. Required source: github.com/microsoft/playwright/releases. Task: On the microsoft/playwright GitHub Releases page, which latest release tag is listed?", "foll...</td><td class="claim"></td><td class="truth">v1.60.0<div class="muted" style="font-size:11px;margin-top:4px">Evidence: v1.60.0</div></td><td><div class="failure">no answer</div><div class="muted" style="font-size:11px;margin-top:3px">Oracle: verified</div></td><td class="source"><div><span class="required">Required:</span> github.com/microsoft/playwright/releases/latest</div><div class="citation-list"><span class="required">Cited:</span><div>github.com/microsoft/playwright/releases</div></div></td></tr>
<tr><td><span class="fail">FAIL</span></td><td><code>stackfit_known_react_default_branch</code><div class="question">On the facebook/react GitHub repository page, which default branch is listed?</div></td><td class="model">http://127.0.0.1:3013/api/agent</td><td class="answer">{"query": "Return the exact single value for this source-specific task. Do not abbreviate numbers. Required source: github.com/facebook/react. Task: On the facebook/react GitHub repository page, which default branch is listed?", "follow_up_questions": null,...</td><td class="claim"></td><td class="truth">main<div class="muted" style="font-size:11px;margin-top:4px">Evidence: main</div></td><td><div class="failure">no answer</div><div class="muted" style="font-size:11px;margin-top:3px">Oracle: verified</div></td><td class="source"><div><span class="required">Required:</span> github.com/facebook/react</div><div class="citation-list"><span class="required">Cited:</span><div>github.com/facebook/react</div></div></td></tr>
<tr><td><span class="fail">FAIL</span></td><td><code>stackfit_known_rust_license_spdx</code><div class="question">On the rust-lang/rust GitHub repository page, which SPDX license identifier is listed?</div></td><td class="model">http://127.0.0.1:3013/api/agent</td><td class="answer">{"query": "Return the exact single value for this source-specific task. Do not abbreviate numbers. Required source: github.com/rust-lang/rust. Task: On the rust-lang/rust GitHub repository page, which SPDX license identifier is listed?", "follow_up_question...</td><td class="claim"></td><td class="truth">Apache-2.0<div class="muted" style="font-size:11px;margin-top:4px">Evidence: Apache-2.0</div></td><td><div class="failure">no answer</div><div class="muted" style="font-size:11px;margin-top:3px">Oracle: verified</div></td><td class="source"><div><span class="required">Required:</span> github.com/rust-lang/rust</div><div class="citation-list"><span class="required">Cited:</span><div>github.com/rust-lang/rust</div></div></td></tr>
<tr><td><span class="pass">PASS</span></td><td><code>stackfit_known_openai_python_client_class</code><div class="question">In the main usage example from the openai-python README on GitHub, which client class appears after `from openai import`?</div></td><td class="model">http://127.0.0.1:3013/api/agent</td><td class="answer">{"query": "Return the exact single value for this source-specific task. Do not abbreviate numbers. Required source: github.com/openai/openai-python. Task: In the main usage example from the openai-python README on GitHub, which client class appears after `f...</td><td class="claim">OpenAI</td><td class="truth">OpenAI<div class="muted" style="font-size:11px;margin-top:4px">Evidence: rom PyPI pip install openai ``` ## Usage The full API of this library can be found in [api.md](api.md). The primary API for interacting w...</div></td><td><div class="failure">correct</div><div class="muted" style="font-size:11px;margin-top:3px">Oracle: verified</div></td><td class="source"><div><span class="required">Required:</span> github.com/openai/openai-python</div><div class="citation-list"><span class="required">Cited:</span><div>github.com/openai/openai-python</div></div></td></tr></tbody>
</table>
<section class="method">
<div class="method-card">
<strong>Oracle values</strong>
<p>Values came from task-pack oracle sources for github.com; each task row lists the required source used for grading.</p>
</div>
<div class="method-card">
<strong>Grading</strong>
<p>5 text tasks normalized case and punctuation before comparison.</p>
</div>
<div class="method-card">
<strong>Source check</strong>
<p>Each citation URL is checked against the task's required source. Matching values from the wrong source are marked as source failures.</p>
</div>
</section>
<section class="insight">
<strong>Result</strong>
<p>4 tasks had issues: stale or incorrect values, missing answers. Examples: stackfit_known_playwright_release_tag, stackfit_known_react_default_branch.</p>
</section>
<section class="split">
<div>
<h2>Outcome Breakdown</h2>
<table><thead><tr><th>Outcome</th><th>Runs</th><th>Share</th></tr></thead><tbody><tr><td><span class="status-dot no-answer"></span>no answer</td><td>3</td><td>60%</td></tr>
<tr><td><span class="status-dot wrong-value"></span>wrong value</td><td>1</td><td>20%</td></tr>
<tr><td><span class="status-dot source-supported-correct"></span>correct with source support</td><td>1</td><td>20%</td></tr></tbody></table>
<h2>Oracle Health</h2>
<table><thead><tr><th>Status</th><th>Runs</th><th>Share</th></tr></thead><tbody><tr><td><span class="status-dot verified"></span>verified</td><td>5</td><td>100%</td></tr></tbody></table>
<h2>Representative Catches</h2>
<div class="catch-list"><div class="catch"><div class="catch-title"><code>stackfit_known_playwright_release_tag</code><span class="failure">no answer</span></div><div class="catch-grid"><div><div class="mini-label">Agent said</div><div class="claim"></div></div><div><div class="mini-label">Oracle had</div><div class="truth">v1.60.0</div></div></div><p class="muted" style="margin-top:8px;font-size:12px">Required source: github.com/microsoft/playwright/releases/latest</p><p class="muted" style="margin-top:3px;font-size:12px">Agent cited: github.com</p></div>
<div class="catch"><div class="catch-title"><code>stackfit_known_react_default_branch</code><span class="failure">no answer</span></div><div class="catch-grid"><div><div class="mini-label">Agent said</div><div class="claim"></div></div><div><div class="mini-label">Oracle had</div><div class="truth">main</div></div></div><p class="muted" style="margin-top:8px;font-size:12px">Required source: github.com/facebook/react</p><p class="muted" style="margin-top:3px;font-size:12px">Agent cited: github.com</p></div>
<div class="catch"><div class="catch-title"><code>stackfit_known_next_package_manager</code><span class="failure">wrong value</span></div><div class="catch-grid"><div><div class="mini-label">Agent said</div><div class="claim">npm</div></div><div><div class="mini-label">Oracle had</div><div class="truth">pnpm@10.33.0</div></div></div><p class="muted" style="margin-top:8px;font-size:12px">Required source: github.com/vercel/next.js/blob/canary/package.json</p><p class="muted" style="margin-top:3px;font-size:12px">Agent cited: github.com, github.com</p></div></div>
</div>
<div>
<h2>Task-Level Results</h2>
<table><thead><tr><th>Task</th><th>Exact</th><th>Source-supported</th><th>Observed outcomes</th></tr></thead><tbody><tr><td><code>stackfit_known_next_package_manager</code><div class="question">In the canary Next.js package.json file on GitHub, what exact packageManager value is listed?</div></td><td>0/1</td><td>0/1</td><td>wrong value</td></tr>
<tr><td><code>stackfit_known_openai_python_client_class</code><div class="question">In the main usage example from the openai-python README on GitHub, which client class appears after `from openai import`?</div></td><td>1/1</td><td>1/1</td><td>correct</td></tr>
<tr><td><code>stackfit_known_playwright_release_tag</code><div class="question">On the microsoft/playwright GitHub Releases page, which latest release tag is listed?</div></td><td>0/1</td><td>0/1</td><td>no answer</td></tr>
<tr><td><code>stackfit_known_react_default_branch</code><div class="question">On the facebook/react GitHub repository page, which default branch is listed?</div></td><td>0/1</td><td>0/1</td><td>no answer</td></tr>
<tr><td><code>stackfit_known_rust_license_spdx</code><div class="question">On the rust-lang/rust GitHub repository page, which SPDX license identifier is listed?</div></td><td>0/1</td><td>0/1</td><td>no answer</td></tr></tbody></table>
</div>
</section>
<footer>Generated from Agent Royale run data.</footer>
</main>
</body>
</html>