Skip to content

Fixes for IDP tools #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thisrohangupta merged 5 commits into from
May 11, 2026
+411 −94
Merged

Fixes for IDP tools #159

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6e3f9e5
Fixes for IDP tools
harjas27 May 8, 2026
2c2345c
Fixes tests
harjas27 May 9, 2026
e779332
Fixes docs
harjas27 May 10, 2026
919df09
Resolve Conflicts
harjas27 May 10, 2026
8ca43b7
Fix README
harjas27 May 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 17 additions & 18 deletions docs/testing/idp_score/test_plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,33 @@
# Test Plan: IDP Score (`idp_score`)
# Test Plan: IDP Score (`idp_score`, `idp_score_summary`)

| Field | Value |
|-------|-------|
| **Resource Type** | `idp_score` |
| **Display Name** | IDP Score |
| **Resource Type** | `idp_score`, `idp_score_summary` |
| **Display Name** | IDP Score, IDP Score Summary |
| **Toolset** | idp |
| **Scope** | account |
| **Operations** | list, get |
| **Operations** | `idp_score`: list; `idp_score_summary`: get |
| **Execute Actions** | None |
| **Identifier Fields** | entity_id |
| **Filter Fields** | None |
| **Identifier Fields** | entity_identifier |
| **Filter Fields** | entity_identifier |
| **Deep Link** | No |

## Test Cases

| Test ID | Category | Description | Prompt | Expected Result |
|---------|----------|-------------|--------|-----------------|
| TC-idp_score-001 | List | List all entity scores with defaults | `harness_list(resource_type="idp_score")` | Returns paginated list of entity scores |
| TC-idp_score-002 | List | List with pagination page 0 | `harness_list(resource_type="idp_score", page=0, size=5)` | Returns first page with up to 5 scores |
| TC-idp_score-003 | List | List with pagination page 1 | `harness_list(resource_type="idp_score", page=1, size=5)` | Returns second page of scores |
| TC-idp_score-004 | List | List with large page size | `harness_list(resource_type="idp_score", size=100)` | Returns up to 100 scores |
| TC-idp_score-005 | Get | Get score by entity_id | `harness_get(resource_type="idp_score", entity_id="my-service")` | Returns score summary for the entity |
| TC-idp_score-006 | Get | Verify score response structure | `harness_get(resource_type="idp_score", entity_id="my-service")` | Response contains score summary fields |
| TC-idp_score-007 | Error | Get with missing entity_id | `harness_get(resource_type="idp_score")` | Error: entity_id is required |
| TC-idp_score-008 | Error | Get non-existent entity score | `harness_get(resource_type="idp_score", entity_id="nonexistent")` | Error: entity not found (404) |
| TC-idp_score-009 | Edge | List with page beyond data | `harness_list(resource_type="idp_score", page=9999)` | Returns empty list |
| TC-idp_score-010 | Edge | List with size=1 | `harness_list(resource_type="idp_score", size=1)` | Returns exactly 1 score |
| TC-idp_score-001 | List | List scorecard scores for an entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/my-service"})` | Returns overall_score plus scorecard score items for the entity |
| TC-idp_score-002 | List | List scorecard scores using top-level entity_identifier | `harness_list(resource_type="idp_score", entity_identifier="default/Component/my-service")` | Returns overall_score plus scorecard score items for the entity |
Copy link
Copy Markdown
Contributor

@cursor cursor Bot May 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

harness_list's public schema only exposes resource-specific inputs via filters / params; entity_identifier is not a declared top-level input in src/tools/harness-list.ts. Documenting the top-level form here makes the test plan rely on an undocumented path, while the supported filters={"entity_identifier": ...} form is already covered by TC-idp_score-001. The test report/README should stay aligned with the published tool contract.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot May 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public harness_list contract does not expose entity_identifier as a top-level argument; resource-specific fields are documented under filters/params in src/tools/harness-list.ts. This example therefore drifts from the runtime/tool-schema surface. Please move this case under filters (or params if that is the intended escape hatch), and make the same fix in test_report.md.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot May 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared harness_list contract does not expose resource-specific fields like entity_identifier at the top level. src/tools/harness-list.ts only merges resource-specific values from filters and params, so this example is documenting an unsupported invocation shape. The same mismatch is repeated in docs/testing/idp_score/test_report.md.

| TC-idp_score-003 | Error | List without entity_identifier | `harness_list(resource_type="idp_score")` | Error from IDP API or validation indicating entity_identifier is required |
| TC-idp_score-004 | Error | List scorecard scores for non-existent entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/nonexistent"})` | Error: entity not found or empty/no scores depending on API behavior |
| TC-idp_score_summary-001 | Get | Get aggregate score summary for an entity | `harness_get(resource_type="idp_score_summary", resource_id="default/Component/my-service")` | Returns aggregate score summary for the entity |
| TC-idp_score_summary-002 | Get | Get aggregate score summary via params | `harness_get(resource_type="idp_score_summary", params={"entity_identifier":"default/Component/my-service"})` | Returns aggregate score summary for the entity |
| TC-idp_score_summary-003 | Error | Get summary without entity_identifier | `harness_get(resource_type="idp_score_summary")` | Error: entity_identifier is required |
| TC-idp_score_summary-004 | Error | Get summary for non-existent entity | `harness_get(resource_type="idp_score_summary", resource_id="default/Component/nonexistent")` | Error: entity not found or empty/no summary depending on API behavior |

## Notes
- Account-scoped resource
- API paths: GET `/v1/scores` (list), GET `/v1/scores/{entityId}` (get)
- No filter fields; only pagination params supported for list
- API paths: GET `/v1/scores` (`idp_score` list), GET `/v1/scores/summary` (`idp_score_summary` get)
- `idp_score` requires `entity_identifier` and does not expose pagination
- `idp_score_summary` requires `entity_identifier`
- No deep link template defined
28 changes: 13 additions & 15 deletions docs/testing/idp_score/test_report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Test Report: IDP Score (`idp_score`)
# Test Report: IDP Score (`idp_score`, `idp_score_summary`)

| Field | Value |
|-------|-------|
| **Resource Type** | `idp_score` |
| **Resource Type** | `idp_score`, `idp_score_summary` |
| **Date** | 2026-03-23 |
| **Tester** | MCP Automated Test |
| **Account ID** | px7xd_BFRCi-pfWPYXVjvw |
Expand All @@ -13,26 +13,24 @@

| Test ID | Description | Prompt | Expected Result | Status | Actual Result | Notes |
|---------|-------------|--------|-----------------|--------|---------------|-------|
| TC-idp_score-001 | List all entity scores with defaults | `harness_list(resource_type="idp_score")` | Returns paginated list of entity scores | ✅ Passed | Returns entity scores (requires entity_identifier filter) | Requires entity_identifier filter |
| TC-idp_score-002 | List with pagination page 0 | `harness_list(resource_type="idp_score", page=0, size=5)` | Returns first page with up to 5 scores | ⬜ Pending | | |
| TC-idp_score-003 | List with pagination page 1 | `harness_list(resource_type="idp_score", page=1, size=5)` | Returns second page of scores | ⬜ Pending | | |
| TC-idp_score-004 | List with large page size | `harness_list(resource_type="idp_score", size=100)` | Returns up to 100 scores | ⬜ Pending | | |
| TC-idp_score-005 | Get score by entity_id | `harness_get(resource_type="idp_score", entity_id="my-service")` | Returns score summary for the entity | ⬜ Pending | | |
| TC-idp_score-006 | Verify score response structure | `harness_get(resource_type="idp_score", entity_id="my-service")` | Response contains score summary fields | ⬜ Pending | | |
| TC-idp_score-007 | Get with missing entity_id | `harness_get(resource_type="idp_score")` | Error: entity_id is required | ⬜ Pending | | |
| TC-idp_score-008 | Get non-existent entity score | `harness_get(resource_type="idp_score", entity_id="nonexistent")` | Error: entity not found (404) | ⬜ Pending | | |
| TC-idp_score-009 | List with page beyond data | `harness_list(resource_type="idp_score", page=9999)` | Returns empty list | ⬜ Pending | | |
| TC-idp_score-010 | List with size=1 | `harness_list(resource_type="idp_score", size=1)` | Returns exactly 1 score | ⬜ Pending | | |
| TC-idp_score-001 | List scorecard scores for an entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/my-service"})` | Returns overall_score plus scorecard score items for the entity | ⬜ Pending | | |
| TC-idp_score-002 | List scorecard scores using top-level entity_identifier | `harness_list(resource_type="idp_score", entity_identifier="default/Component/my-service")` | Returns overall_score plus scorecard score items for the entity | ⬜ Pending | | |
| TC-idp_score-003 | List without entity_identifier | `harness_list(resource_type="idp_score")` | Error from IDP API or validation indicating entity_identifier is required | ⬜ Pending | | |
| TC-idp_score-004 | List scorecard scores for non-existent entity | `harness_list(resource_type="idp_score", filters={"entity_identifier":"default/Component/nonexistent"})` | Error: entity not found or empty/no scores depending on API behavior | ⬜ Pending | | |
| TC-idp_score_summary-001 | Get aggregate score summary for an entity | `harness_get(resource_type="idp_score_summary", resource_id="default/Component/my-service")` | Returns aggregate score summary for the entity | ⬜ Pending | | |
| TC-idp_score_summary-002 | Get aggregate score summary via params | `harness_get(resource_type="idp_score_summary", params={"entity_identifier":"default/Component/my-service"})` | Returns aggregate score summary for the entity | ⬜ Pending | | |
| TC-idp_score_summary-003 | Get summary without entity_identifier | `harness_get(resource_type="idp_score_summary")` | Error: entity_identifier is required | ⬜ Pending | | |
| TC-idp_score_summary-004 | Get summary for non-existent entity | `harness_get(resource_type="idp_score_summary", resource_id="default/Component/nonexistent")` | Error: entity not found or empty/no summary depending on API behavior | ⬜ Pending | | |

## Summary

| Metric | Count |
|--------|-------|
| Total Tests | 10 |
| ✅ Passed | 1 |
| Total Tests | 8 |
| ✅ Passed | 0 |
| ❌ Failed | 0 |
| ⚠️ Blocked | 0 |
| ⬜ Not Run | 9 |
| ⬜ Not Run | 8 |

## Issues Found

Expand Down
Loading
Loading