Skip to content

Fixes for IDP tools#159

Merged
thisrohangupta merged 5 commits intomainfrom
idp_tools
May 11, 2026
Merged

Fixes for IDP tools#159
thisrohangupta merged 5 commits intomainfrom
idp_tools

Conversation

@harjas27
Copy link
Copy Markdown
Contributor

@harjas27 harjas27 commented May 8, 2026

Description

  • Fixes the IDP get_entity
  • Improves the tool operations for IDP entities by supporting more parameters for searching and listing

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • Other

Checklist

  • Tests pass
  • Typecheck passes

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Collaborator

@thisrohangupta thisrohangupta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Checks are failing can you review?

Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Found 3 warnings against Sunil's architecture standards:

  1. src/registry/toolsets/idp.ts: idp_workflow.execute has a metadata/runtime mismatch. workflow_details is mandatory in bodyBuilder(), but it is not represented in bodySchema.fields, so harness_describe exposes an incomplete contract and the shared required-field validation cannot catch the real missing input. This repo's standards prefer structured agent guidance over prose-only actionDescription.
  2. src/registry/toolsets/idp.ts: scorecard.list now stops forwarding page/size and claims the endpoint has no pagination. harness_list still accepts pagination inputs, so callers now get silent no-op behavior instead of consistent shared-list semantics or a fail-loud error.
  3. Public docs are no longer in sync with the runtime surface. pnpm build && pnpm docs:check fails with README.md is stale, and the idp_score split leaves stale get/pagination docs in README.md and docs/testing/idp_score/*.

Assumptions / questions:

  • I assumed the existing scorecard and idp_score docs still reflect the intended public contract. If the underlying IDP APIs changed, those docs need to be updated in the same PR.
  • I did not see focused tests covering the new idp_workflow.execute request-shaping path, so there is residual risk around IDP response/body contract assumptions.

Checks run:

  • gh pr checks 159
  • pnpm build
  • pnpm test
  • pnpm typecheck
  • pnpm build && pnpm docs:check (fails: README.md is stale)
Open in Web View Automation 

Sent by Cursor Automation: Sunil On Demand Architecture Review

Comment thread src/registry/toolsets/idp.ts Outdated
Comment thread src/registry/toolsets/idp.ts
Comment thread src/registry/toolsets/idp.ts
Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Re-ran the architecture review on the latest head. The earlier docs:check failure is fixed, but I still found 4 issues against Sunil's standards:

  1. idp_workflow.execute still promises local validation of required spec.parameters[], but the implementation only parses auth-token refs from spec.steps[] and never checks missing non-auth params before sending the request.
  2. workflow_details is still required only in prose/custom code while bodySchema marks it optional, so harness_describe exposes an incomplete execute contract.
  3. The new IDP scope helpers synthesize placeholder scopes such as account.org.project when explicit org/project scope is requested without ids instead of failing locally with a clear missing-scope error.
  4. Public docs are still partially out of sync with the runtime surface: README.md still shows idp_score as get-capable and omits idp_score_summary, and the updated idp_score test plan/report document an unsupported top-level entity_identifier call shape for harness_list.

Assumptions:

  • I assumed the published harness_list schema in src/tools/harness-list.ts is the supported contract for top-level inputs.

Checks run:

  • pnpm install --frozen-lockfile
  • pnpm typecheck
  • pnpm test
  • pnpm build && pnpm docs:check
Open in Web View Automation 

Sent by Cursor Automation: Sunil On Demand Architecture Review

}
}

const requestBody = { identifier, values };
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The action contract now says values are validated against required workflow_details.yaml spec.parameters, but bodyBuilder() never inspects spec.parameters[] before returning { identifier, values }. Today we only auto-fill auth-token refs from spec.steps[], so missing required non-auth params will fail downstream in IDP instead of locally. That conflicts with the repo's fail-loud preference and makes harness_describe promise behavior we don't implement.

{
name: "workflow_details",
type: "object",
required: false,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_details is mandatory for this action to build the request, but the structured schema marks it required: false. Because Registry.executeSpec() only uses bodySchema.fields for shared missing-field validation, harness_describe exposes an incomplete contract and callers get a late custom throw from bodyBuilder() instead of the standard schema error. Per the repo's "prefer data over prose" rule, this should be modeled structurally rather than only described in prose.

case "project":
if (orgId && projectId) scopes = `account.${orgId}.${projectId}`;
else if (orgId) scopes = `account.${orgId}.project`;
else scopes = "account.org.project";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a caller explicitly picks scope_level=org or project without the matching IDs, this builds placeholder scopes like account.org / account.org.project and punts the error to the IDP API. Sunil's standards prefer fail-loud local validation over synthetic identifiers. Can we throw here (and in the mirrored idp_entity pathBuilder above) unless the needed org_id / project_id is present?

| TC-idp_score-009 | Edge | List with page beyond data | `harness_list(resource_type="idp_score", page=9999)` | Returns empty list |
| TC-idp_score-010 | Edge | List with size=1 | `harness_list(resource_type="idp_score", size=1)` | Returns exactly 1 score |
| TC-idp_score-001 | List | List scorecard scores for an entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/my-service"})` | Returns overall_score plus scorecard score items for the entity |
| TC-idp_score-002 | List | List scorecard scores using top-level entity_identifier | `harness_list(resource_type="idp_score", entity_identifier="default/Component/my-service")` | Returns overall_score plus scorecard score items for the entity |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

harness_list's public schema only exposes resource-specific inputs via filters / params; entity_identifier is not a declared top-level input in src/tools/harness-list.ts. Documenting the top-level form here makes the test plan rely on an undocumented path, while the supported filters={"entity_identifier": ...} form is already covered by TC-idp_score-001. The test report/README should stay aligned with the published tool contract.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment
  1. Important — src/registry/toolsets/idp.ts: idp_workflow.execute still promises local validation of required workflow_details.yaml.spec.parameters[], but bodyBuilder() only inspects spec.steps[] for auth-token refs and never rejects missing non-auth required params before POSTing to /v2/workflows/execute. This is still not fail-loud, and the runtime behavior is looser than the published execute contract.
  2. Important — src/registry/toolsets/idp.ts: workflow_details is still required in bodyBuilder()/actionDescription, but bodySchema.fields marks it optional. harness_describe therefore advertises the wrong contract, and the shared required-field validation path cannot catch the real missing input.
  3. Important — src/registry/toolsets/idp.ts: the new scope_level helpers still synthesize placeholders such as account.org, account.org.project, and account.<org>.project when explicit org/project scope is requested without the corresponding ids. That turns a missing-scope input into an opaque remote lookup instead of a clear local org_id/project_id error. The same bug appears in both idp_entity.list and idp_workflow.list.
  4. Important — docs/testing/idp_score/test_plan.md, docs/testing/idp_score/test_report.md, README.md: docs/runtime parity is still off. The changed idp_score docs still show unsupported top-level entity_identifier calls for harness_list, and pnpm build && pnpm docs:check still fails because README.md is stale (idp_score is still shown as get-capable and idp_score_summary is omitted).

Previously reported issue that does appear fixed on the current head: scorecard.list is forwarding page/size again, so the earlier pagination regression there is gone.

Open in Web View Automation 

Sent by Cursor Automation: Sunil On Demand Architecture Review

);
}

const refs = extractAuthParamRefs(yamlStr);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actionDescription says this call validates the required spec.parameters[], but this implementation only inspects spec.steps[] via extractAuthParamRefs(). Missing non-auth required parameters will still fall through to the remote /v2/workflows/execute call instead of failing locally. Please parse workflow_details.yaml.spec.parameters here and reject any required non-auth keys missing from values before building the request.

{
name: "workflow_details",
type: "object",
required: false,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_details is required above (if (!wfDetails) throw ...) and in actionDescription, but the published bodySchema still marks it optional. Because registry preflight only enforces bodySchema.required, harness_describe exposes the wrong execute contract and the shared validation path cannot catch the missing field. Mark this as required or relax the runtime requirement so metadata and behavior stay in sync.

Comment on lines +90 to +96
case "org":
scopes = orgId ? `account.${orgId}` : "account.org";
break;
case "project":
if (orgId && projectId) scopes = `account.${orgId}.${projectId}`;
else if (orgId) scopes = `account.${orgId}.project`;
else scopes = "account.org.project";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still synthesizes placeholder scopes like account.org, account.org.project, and account.${orgId}.project when callers request explicit org/project scope without supplying the needed ids. That converts a local configuration/input error into an opaque remote lookup and violates the repo's fail-loud scope handling. Please throw here (and in the duplicated workflow pathBuilder below) when scope_level requires org_id/project_id that are unavailable.

| TC-idp_score-009 | Edge | List with page beyond data | `harness_list(resource_type="idp_score", page=9999)` | Returns empty list |
| TC-idp_score-010 | Edge | List with size=1 | `harness_list(resource_type="idp_score", size=1)` | Returns exactly 1 score |
| TC-idp_score-001 | List | List scorecard scores for an entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/my-service"})` | Returns overall_score plus scorecard score items for the entity |
| TC-idp_score-002 | List | List scorecard scores using top-level entity_identifier | `harness_list(resource_type="idp_score", entity_identifier="default/Component/my-service")` | Returns overall_score plus scorecard score items for the entity |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public harness_list contract does not expose entity_identifier as a top-level argument; resource-specific fields are documented under filters/params in src/tools/harness-list.ts. This example therefore drifts from the runtime/tool-schema surface. Please move this case under filters (or params if that is the intended escape hatch), and make the same fix in test_report.md.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current HEAD is in better shape than the earlier revisions, but I still found 3 important issues in src/registry/toolsets/idp.ts against the repo standards in CLAUDE.md / AGENTS.md:

  1. Important — src/registry/toolsets/idp.ts:90-96, 383-389
    idp_entity.list and idp_workflow.list still synthesize placeholder scopes like account.org, account.org.project, and account.${orgId}.project when callers explicitly request scope_level="org" / "project" without the required identifiers. That turns a local input/config problem into an opaque remote lookup instead of failing loudly with a clear org_id / project_id validation error.

  2. Important — src/registry/toolsets/idp.ts:466-499, 503-513, 531-535
    idp_workflow.execute still does not validate required workflow_details.yaml.spec.parameters[] before POSTing to /v2/workflows/execute. The implementation only parses auth-token refs from spec.steps[], so missing non-auth required inputs are deferred to the remote API. That conflicts with both the published execute contract and the repo’s fail-loud validation standard.

  3. Important — src/registry/toolsets/idp.ts:444-447, 518-523
    workflow_details is still a hard runtime requirement, but the structured bodySchema publishes it as optional. Because harness_describe surfaces bodySchema and shared validation only knows about structured fields, the real execute contract is still encoded partly in ad hoc prose/custom throws instead of accurate structured metadata.

Previously reported issues that do appear fixed on current HEAD:

  • src/registry/toolsets/idp.ts:187-190 now forwards page / size again for scorecard.list, so the earlier pagination regression is gone.
  • The earlier repo-check failure no longer reproduces here: pnpm typecheck, pnpm build && pnpm docs:check, and pnpm test -- --runInBand all pass.
  • workflow_details is now present in bodySchema (earlier it was missing entirely), but that is only a partial fix because the required flag is still wrong, so I kept the contract-alignment finding above.
Open in Web View Automation 

Sent by Cursor Automation: Sunil On Demand Architecture Review

case "project":
if (orgId && projectId) scopes = `account.${orgId}.${projectId}`;
else if (orgId) scopes = `account.${orgId}.project`;
else scopes = "account.org.project";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When callers explicitly ask for scope_level="org" / "project" without the matching ids, this builds placeholder scopes like account.org, account.org.project, and account.${orgId}.project instead of failing locally. That violates the repo’s fail-loud rule and makes the error depend on a remote lookup. Please throw a clear org_id / project_id validation error here, and make the mirrored idp_workflow.list pathBuilder below do the same.

);
}

const refs = extractAuthParamRefs(yamlStr);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actionDescription / bodySchema say values are validated against required workflow_details.yaml.spec.parameters[], but this code only parses auth-token refs from spec.steps[] and then POSTs whatever values contains. Missing non-auth required params still fail downstream in IDP instead of locally, which conflicts with the repo’s fail-loud validation standard. Please parse spec.parameters here and reject missing required non-auth keys before building the request.

{
name: "workflow_details",
type: "object",
required: false,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_details is still a hard runtime requirement (if (!wfDetails) throw ... above), but the structured schema publishes it as optional. Since harness_describe surfaces bodySchema and registry preflight only understands structured fields, this keeps the real execute contract in ad hoc prose/custom errors instead of data. Please make the structured contract and runtime requirement match, either by adding explicit pre-bodyBuilder validation for auxiliary execute inputs like workflow_details, or by removing the runtime requirement.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran the architecture review on the current head. Local validation is green (pnpm install --frozen-lockfile, pnpm typecheck, pnpm build && pnpm docs:check, pnpm test -- --runInBand), but I still found 4 important issues against Sunil's standards:

  1. idp_entity.list and idp_workflow.list still synthesize placeholder scopes like account.org, account.org.project, and account.<org>.project when callers request explicit org/project scope without the needed ids. That turns a missing-scope input into an opaque remote query instead of a fail-loud local org_id / project_id error.
  2. idp_workflow.execute still promises local validation of required workflow_details.yaml spec.parameters[], but the implementation only inspects spec.steps[] for auth-token refs and never rejects missing non-auth required params before POSTing.
  3. workflow_details is still runtime-required but modeled as optional in the structured execute schema, so harness_describe advertises the wrong contract and the shared schema-based validation path cannot catch the real missing field.
  4. Docs/runtime parity is still off. docs/testing/idp_score/* still documents an unsupported top-level entity_identifier shape for harness_list, and README.md still shows idp_score as get-capable while omitting idp_score_summary from the IDP surface.

Previously reported issue that does appear fixed on the current head: scorecard.list is forwarding pagination inputs again, and the repo checks are now passing.

Open in Web View Automation 

Sent by Cursor Automation: Sunil On Demand Architecture Review

Comment on lines +90 to +96
case "org":
scopes = orgId ? `account.${orgId}` : "account.org";
break;
case "project":
if (orgId && projectId) scopes = `account.${orgId}.${projectId}`;
else if (orgId) scopes = `account.${orgId}.project`;
else scopes = "account.org.project";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the caller explicitly requests scope_level="org" or scope_level="project", falling back to placeholders like account.org / account.org.project is not a valid substitute for the missing identifiers. Under this repo's Fail Loudly rule, this should reject locally with a clear org_id / project_id error instead of turning a missing-input bug into an opaque remote lookup.

The same placeholder-scope pattern is duplicated again in idp_workflow.list below.

Comment on lines +466 to +490
const refs = extractAuthParamRefs(yamlStr);
const values = { ...((b.values as Record<string, unknown> | undefined) ?? {}) };

for (const ref of refs.apikeyRefs) {
if (values[ref] === undefined) values[ref] = "user.token";
}

if (refs.apiKeySecretRefs.length > 0) {
const userSupplied =
(b.api_key_secret as string | undefined) ??
(input.api_key_secret as string | undefined);
const fallback = input[CONFIG_API_KEY] as string | undefined;
const keyValue = userSupplied || fallback;
if (!keyValue) {
throw new Error(
"Missing apiKeySecret. This workflow has a step with an apiKeySecret input but no api_key_secret was provided and HARNESS_API_KEY is not configured. Pass apiKeySecret in the body.",
);
}

for (const ref of refs.apiKeySecretRefs) {
if (values[ref] === undefined) values[ref] = keyValue;
}
}

const requestBody = { identifier, values };
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The execute contract and actionDescription both say values should be validated against required workflow_details.yaml spec.parameters, but this builder only extracts auth-token refs from spec.steps[] and auto-injects those values. Missing non-auth required params still flow through to /v2/workflows/execute, so the implementation is looser than the published contract and no longer fails loudly before the write.

{
name: "workflow_details",
type: "object",
required: false,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_details is runtime-required a few lines above, but the structured schema still marks it optional here. Because harness_describe and the shared required-field validation path both derive from bodySchema, agents see an inaccurate execute contract and the missing field is only caught by custom bodyBuilder() logic. That goes against the repo rule to prefer structured metadata over prose-only guidance.

| TC-idp_score-009 | Edge | List with page beyond data | `harness_list(resource_type="idp_score", page=9999)` | Returns empty list |
| TC-idp_score-010 | Edge | List with size=1 | `harness_list(resource_type="idp_score", size=1)` | Returns exactly 1 score |
| TC-idp_score-001 | List | List scorecard scores for an entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/my-service"})` | Returns overall_score plus scorecard score items for the entity |
| TC-idp_score-002 | List | List scorecard scores using top-level entity_identifier | `harness_list(resource_type="idp_score", entity_identifier="default/Component/my-service")` | Returns overall_score plus scorecard score items for the entity |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared harness_list contract does not expose resource-specific fields like entity_identifier at the top level. src/tools/harness-list.ts only merges resource-specific values from filters and params, so this example is documenting an unsupported invocation shape. The same mismatch is repeated in docs/testing/idp_score/test_report.md.

@thisrohangupta thisrohangupta merged commit ba128f0 into main May 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants