Skip to content

Commit 36dfc5a

Browse files
haz3141haz3141
authored
v0.6.4 — RAG Evaluation Gates (MVP) (#33)
* docs(changelog): note Cursor MCP audit and CI guardrails - Added comprehensive audit and hardening of IDE/MCP integration - Documented MCP server health endpoints and VS Code configuration - Noted CI guardrails and evidence documentation - Fixed pre-commit configuration and security issues * docs(freeze): declare code freeze for v0.6.3 (NY time) * chore(version): bump to v0.6.3 * docs(changelog): finalize v0.6.3 (NY) and roll Unreleased forward * docs(evidence): v0.6.3 index (NY) * chore(eval): step 7 scaffolding stubs only (no dataset) * chore(cursor): isolate MCP to stdio baseline; pin interpreter and env - Single active .cursor/mcp.json entry with allowlist - Stdio MCP via -m mcp_server.simple_server - Logs to stderr, zero stdout - Add conservative settings/environment defaults - Add freeze guardrails and terminal memory system - Lower temperature to 0.1 for determinism * feat(mcp): register baseline ping tool in stdio server - Minimal FastMCP app.run(transport="stdio") - ping(message) -> str; deterministic echo - No stdout noise; structured responses - Proper error handling and logging to stderr * feat(mcp): add summarize tool (stub); wire DSPy later - Validates registration + JSON shape - Fallback summarization when DSPy unavailable - Configurable max_length parameter - Next: replace stub with DSPy summarizer module * chore: clean up temporary files and add legacy MCP server - Remove .coverage, package-lock.json, package.json - Add legacy mcp_server.py for reference - Clean working tree for freeze compliance * chore(cursor): configure grok-code-fast-1 max mode; clamp to read-only * refactor(mcp): unify FastMCP into single app instance; switch tools to register(app) - Create single FastMCP app instance in mcp_server/app.py - Convert all tools to register(app) pattern to avoid API drift - Fix simple_server.py to use explicit tool registration - Ensure clean stdout for JSON-RPC stdio transport - All 3 tools (ping, search_docs, summarize) now properly registered * feat: add RAG evaluation gates infrastructure - Add eval/run.py main evaluation runner - Add eval/configs/lab.yaml configuration - Add eval/data/lab/ test datasets - Add scripts/ci/parse_metrics.py gate parser - Add .github/workflows/rag-gates.yml CI integration - Add evidence/learning/ structure for v0.6.4 * fix: cleanup and finalize v0.6.4 implementation - Fixed linting issues in eval/run.py - All gates passing with mock data - MCP server integration working - Configuration files validated - Documentation complete - Ready for production deployment * feat: add RAG evaluation gates infrastructure - Add eval/run.py main evaluation runner - Add eval/configs/lab.yaml configuration - Add eval/data/lab/ test datasets - Add scripts/ci/parse_metrics.py gate parser - Add .github/workflows/rag-gates.yml CI integration - Update eval/README.md with framework documentation * chore: bump version to v0.6.4 - Update VERSION to 0.6.4 - Add v0.6.4 changelog entry with RAG evaluation gates - Document comprehensive evaluation framework and CI integration * fix: correct v0.6.4 RAG evaluation gates implementation - Fix CI workflow to work with actual MCP server architecture - Remove broken HTTP endpoint tests that require authentication - Add proper dependency installation (numpy, scikit-learn) - Add directory creation step for evaluation runs - Test only safe endpoints (health, summarize, audit) - Ensure evaluation pipeline works correctly Fixes PR #33 CI failures * fix: correct MCP allowlist validation to allow underscores in tool names - Updated regex pattern in validate_mcp_allowlist.py to allow underscores - Tool names like 'tools.search_docs' now pass validation - Fixes CI security validation step failures - All CI steps now pass locally * feat: implement comprehensive Cursor project rules - Add 6 properly formatted MDC rules with YAML frontmatter - Always applied: project-guardrails.mdc, security-mcp.mdc - Auto-attached: documentation.mdc (docs/), rag-evaluation.mdc (eval/rag/) - Remove old conflicting rule files - Enable context-aware AI assistance for development workflow * feat: v0.6.4 RAG evaluation gates * fix: update CI workflows and add missing doc headers
1 parent e01638f commit 36dfc5a

65 files changed

Lines changed: 3946 additions & 1631 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.coverage‎

-52 KB
Binary file not shown.

‎.cursor/environment.json‎

Lines changed: 7 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,11 @@
11
{
22
"commands": {
3-
"dev:mcp": "uvicorn mcp_server.server:app --reload --host 127.0.0.1 --port 8765",
4-
"health": "curl -fsS http://127.0.0.1:8765/healthz || curl -fsS http://127.0.0.1:8765/ | head -n 1",
5-
"test": "pytest -q",
6-
"test:security": "pytest lab/security/tests/ -v",
7-
"test:integration": "pytest lab/tests/ -v",
8-
"eval": "python lab/eval/run_eval.py --dataset lab/eval/dataset.jsonl --k 5",
9-
"eval:full": "python lab/eval/run_eval.py --dataset lab/eval/dataset.jsonl --k 5 --output eval_results.json",
10-
"obs:ingest": "python lab/obs/ingest.py --path logs/audit/*.jsonl",
11-
"obs:audit": "python lab/obs/audit.py --recent 100",
12-
"lint": "ruff check .",
13-
"format": "ruff format . && black .",
14-
"format:check": "ruff format --check . && black --check .",
15-
"docs:check": "find docs/ -name '*.md' -exec grep -L '<!-- Version:' {} \\;"
16-
},
17-
"environment": {
18-
"PYTHONPATH": ".",
19-
"LOG_LEVEL": "INFO",
20-
"GUARDIAN_ALLOW_TOOLS": "health,tools/search_docs,tools/summarize"
3+
"pytest": "python -m pytest",
4+
"ruff": "python -m ruff",
5+
"mcp-list": "cursor-agent mcp list",
6+
"mcp-tools": "cursor-agent mcp list-tools lab-server",
7+
"eval": "python eval/run.py --dataset eval/data/lab/lab_dev.jsonl --output eval/runs/$(date +%Y%m%d-%H%M%S)",
8+
"test": "python -m pytest tests/",
9+
"mcp": ".venv/bin/python -m mcp_server.simple_server"
2110
}
2211
}

‎.cursor/mcp.json‎

Lines changed: 8 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,9 @@
1-
[
2-
{
3-
"name": "lab-server",
4-
"url": "http://127.0.0.1:8765/sse",
5-
"method": "sse",
6-
"allowTools": [
7-
"search_docs",
8-
"summarize",
9-
"rag_query",
10-
"run_tests",
11-
"eval_metrics",
12-
"audit_recent",
13-
"audit_by_request",
14-
"audit_by_tool"
15-
],
16-
"timeout": 30,
17-
"retries": 3,
18-
"gracePeriodSec": 2
1+
{
2+
"mcpServers": {
3+
"lab-server": {
4+
"command": ".venv/bin/python",
5+
"args": ["-m", "mcp_server.simple_server"],
6+
"env": {}
7+
}
198
}
20-
]
9+
}

‎.cursor/rules/code-organization.mdc‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
---
2+
description: Code structure and development patterns
3+
globs: ["**/*.{py,js,ts,md}"]
4+
alwaysApply: false
5+
---
6+
7+
# Code Organization - AI-Dev-Lab v0.6.4
8+
9+
## Directory Structure
10+
11+
### Lab vs App Separation
12+
```
13+
lab/ # Experimental development
14+
├── dsp/ # Data science pipelines
15+
├── eval/ # Evaluation frameworks
16+
├── rag/ # RAG system components
17+
├── security/ # Security tools and policies
18+
└── tests/ # Lab-specific tests
19+
20+
app/ # Production-ready components
21+
├── mcp-servers/ # MCP server implementations
22+
└── ... # Production services
23+
```
24+
25+
### Evaluation Pipeline Structure
26+
```
27+
eval/
28+
├── configs/ # Evaluation configurations
29+
├── data/ # Test datasets
30+
├── pipeline/ # Evaluation execution
31+
├── prompts/ # Evaluation prompts
32+
└── runs/ # Evaluation results
33+
```
34+
35+
## MCP Server Architecture
36+
37+
### Server Organization
38+
- **Single Responsibility**: Each server handles one domain
39+
- **Tool Registration**: Tools registered with clear descriptions
40+
- **Health Endpoints**: `/health` endpoint for monitoring
41+
- **Graceful Shutdown**: Proper cleanup on termination
42+
43+
### Tool Design Patterns
44+
```python
45+
# Tool registration pattern
46+
@server.tool()
47+
async def tool_name(args: ToolArgs) -> ToolResult:
48+
"""Clear tool description."""
49+
# Input validation
50+
# Business logic
51+
# Output formatting
52+
return result
53+
```
54+
55+
## Code Standards
56+
57+
### Import Organization (isort)
58+
```python
59+
# Standard library imports
60+
import os
61+
import sys
62+
63+
# Third-party imports
64+
import yaml
65+
from fastapi import FastAPI
66+
67+
# Local imports
68+
from .utils import helper_function
69+
```
70+
71+
### Type Hints and Documentation
72+
- **Function Signatures**: Full type hints required
73+
- **Docstrings**: Google/NumPy style docstrings
74+
- **Return Types**: Explicit return type annotations
75+
- **Parameter Types**: Input parameter type hints
76+
77+
### Error Handling Patterns
78+
```python
79+
try:
80+
result = risky_operation()
81+
except SpecificException as e:
82+
logger.error(f"Operation failed: {e}")
83+
raise CustomError("User-friendly message") from e
84+
```
85+
86+
## Testing Requirements
87+
88+
### Test Coverage Thresholds
89+
- **Minimum Coverage**: 68% overall
90+
- **Critical Paths**: 85% for security-related code
91+
- **New Features**: 80% for new functionality
92+
- **Regression Tests**: Required for bug fixes
93+
94+
### Test Organization
95+
```python
96+
# test_file.py
97+
import pytest
98+
from src.module import function_to_test
99+
100+
class TestFunctionToTest:
101+
def test_success_case(self):
102+
# Arrange
103+
input_data = "test_input"
104+
105+
# Act
106+
result = function_to_test(input_data)
107+
108+
# Assert
109+
assert result == expected_output
110+
111+
def test_error_case(self):
112+
# Arrange & Act & Assert
113+
with pytest.raises(ExpectedException):
114+
function_to_test(invalid_input)
115+
```
116+
117+
### Integration Testing
118+
- **MCP Server Tests**: End-to-end tool testing
119+
- **Evaluation Pipeline Tests**: Full pipeline validation
120+
- **Security Tests**: Penetration testing scenarios
121+
122+
## Performance Guidelines
123+
124+
### Code Efficiency
125+
- **Algorithm Complexity**: Document Big O for critical paths
126+
- **Memory Usage**: Monitor and optimize memory consumption
127+
- **Async Patterns**: Use async/await for I/O operations
128+
- **Caching**: Implement appropriate caching strategies
129+
130+
### Monitoring and Metrics
131+
- **Performance Metrics**: Response times, throughput
132+
- **Error Rates**: Track and alert on error patterns
133+
- **Resource Usage**: CPU, memory, disk monitoring
134+
- **Health Checks**: Automated health validation
135+
136+
## Development Workflow
137+
138+
### Code Review Checklist
139+
- ✅ Type hints present and correct
140+
- ✅ Tests added/updated with sufficient coverage
141+
- ✅ Documentation updated
142+
- ✅ Security review completed
143+
- ✅ Performance impact assessed
144+
145+
### Promotion Criteria (Lab → App)
146+
- ✅ All tests passing
147+
- ✅ Security audit cleared
148+
- ✅ Documentation complete
149+
- ✅ Performance benchmarks met
150+
- ✅ Code review approved

‎.cursor/rules/docs.mdc‎

Lines changed: 0 additions & 17 deletions
This file was deleted.

0 commit comments

Comments
 (0)