Skip to content

Commit 54006d8

Browse files
committed
Migrate Priority 1 documentation from cortex-platform
Architecture (12 files): - 3 ADRs (MoE, pattern-based learning, worker pool) - Core principles framework - Master-worker pattern - Mixture of Experts architecture - ML/AI system architecture - Event-driven architecture - Hybrid routing architecture - Medallion architecture - Automation architecture Operations (15 runbooks): - Daily operations checklist - Emergency recovery procedures - Worker lifecycle and failure handling - Daemon management - Circuit breaker recovery - MoE router troubleshooting - Performance debugging - Task queue management - Token budget management - Data quality issues - Self-healing operations - Pre-deployment testing - And more... Knowledge (11 guides): - Quick start guide - Developer guide - Command cheatsheet - Self-healing system - Auto-learning system - Coordination protocol - Evaluation framework - RAG system implementation - ML/AI quickstart and deployment - Prompting best practices Total: 38 documentation files migrated Source: cortex-platform/docs and cortex-k3s root
1 parent 6595f76 commit 54006d8

38 files changed

Lines changed: 18392 additions & 0 deletions
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# ADR 001: Use Mixture of Experts (MoE) Architecture
2+
3+
**Status**: Accepted
4+
**Date**: 2025-11-01
5+
**Deciders**: System Architects
6+
**Technical Story**: Multi-repository management system
7+
8+
## Context
9+
10+
cortex needs to manage multiple repositories efficiently with specialized expertise for different types of tasks (development, security, CI/CD, inventory management). A single monolithic approach would lack the specialization needed for optimal task execution.
11+
12+
## Decision
13+
14+
We will use a Mixture of Experts (MoE) architecture with specialized master agents:
15+
16+
- **Coordinator Master**: Routes tasks to appropriate specialist masters
17+
- **Development Master**: Handles feature development and bug fixes
18+
- **Security Master**: Manages security scans and vulnerability remediation
19+
- **CI/CD Master**: Orchestrates builds, tests, and deployments
20+
- **Inventory Master**: Catalogs and documents repositories
21+
- **Dashboard Master**: Provides real-time monitoring
22+
23+
## Rationale
24+
25+
### Advantages
26+
27+
1. **Specialization**: Each master has focused expertise
28+
2. **Scalability**: Masters can scale independently
29+
3. **Maintainability**: Clear separation of concerns
30+
4. **Performance**: Parallel task execution
31+
5. **Learning**: Pattern-based routing improves over time
32+
33+
### Example Routing Logic
34+
35+
```javascript
36+
function routeTask(task) {
37+
if (task.description.match(/security|cve|vulnerability/i)) {
38+
return 'security-master';
39+
}
40+
if (task.description.match(/deploy|build|test|ci/i)) {
41+
return 'cicd-master';
42+
}
43+
if (task.description.match(/fix|bug|feature|implement/i)) {
44+
return 'development-master';
45+
}
46+
return 'coordinator-master'; // Default fallback
47+
}
48+
```
49+
50+
## Consequences
51+
52+
### Positive
53+
54+
- Clear task ownership
55+
- Better resource utilization
56+
- Improved success rates through specialization
57+
- Easier to add new masters
58+
59+
### Negative
60+
61+
- Increased system complexity
62+
- Routing overhead
63+
- Potential routing errors
64+
- Need for coordinator logic
65+
66+
### Mitigation
67+
68+
- Confidence scoring for routing decisions
69+
- Fallback to coordinator for uncertain tasks
70+
- Continuous learning from routing outcomes
71+
- Monitoring and alerting on routing confidence
72+
73+
## Implementation
74+
75+
- **Routing Logic**: `coordination/masters/coordinator/lib/moe-router.sh`
76+
- **Master Specs**: `coordination/masters/{master-name}/`
77+
- **Task Queue**: `coordination/tasks/pending/`
78+
- **Decision Log**: `coordination/masters/coordinator/knowledge-base/routing-decisions.jsonl`
79+
80+
## Alternatives Considered
81+
82+
### 1. Monolithic Agent
83+
84+
**Pros**: Simpler architecture
85+
**Cons**: Lacks specialization, harder to scale
86+
87+
### 2. Rule-Based Task Assignment
88+
89+
**Pros**: Predictable routing
90+
**Cons**: Inflexible, no learning capability
91+
92+
### 3. Manual Task Assignment
93+
94+
**Pros**: Full control
95+
**Cons**: Not scalable, requires human intervention
96+
97+
## Related Decisions
98+
99+
- ADR 002: Pattern-Based Learning for Routing
100+
- ADR 003: Worker Pool Management
101+
102+
## References
103+
104+
- [Mixture of Experts Paper](https://arxiv.org/abs/1701.06538)
105+
- [MoE Implementation Guide](https://github.com/ry-ops/cortex/wiki/MoE-Architecture)
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# ADR 002: Pattern-Based Learning for MoE Routing
2+
3+
**Status**: Accepted
4+
**Date**: 2025-11-15
5+
**Deciders**: ML Team, System Architects
6+
7+
## Context
8+
9+
Initial MoE routing used simple keyword matching, resulting in:
10+
- 65% routing accuracy
11+
- No improvement over time
12+
- Inability to handle novel task types
13+
14+
We need routing to improve based on historical outcomes.
15+
16+
## Decision
17+
18+
Implement pattern-based learning that:
19+
20+
1. **Logs all routing decisions** with outcomes
21+
2. **Learns from successful routes** (task completion)
22+
3. **Adjusts confidence scores** based on patterns
23+
4. **Updates routing model** hourly
24+
25+
### Learning Algorithm
26+
27+
```javascript
28+
function updateRoutingModel(decision, outcome) {
29+
const pattern = extractPattern(decision.task);
30+
const success = outcome.status === 'completed';
31+
32+
// Update pattern weights
33+
if (success) {
34+
patterns[pattern].weight += 0.1;
35+
patterns[pattern].confidence = Math.min(patterns[pattern].confidence + 0.05, 1.0);
36+
} else {
37+
patterns[pattern].weight -= 0.05;
38+
patterns[pattern].confidence = Math.max(patterns[pattern].confidence - 0.1, 0);
39+
}
40+
41+
// Save updated patterns
42+
savePatterns(patterns);
43+
}
44+
```
45+
46+
## Rationale
47+
48+
- **Continuous Improvement**: System gets smarter over time
49+
- **Data-Driven**: Based on actual outcomes, not assumptions
50+
- **Adaptive**: Handles new task types automatically
51+
- **Transparent**: All decisions logged for review
52+
53+
## Consequences
54+
55+
### Positive
56+
57+
- Routing accuracy improved from 65% → 85%
58+
- Handles edge cases better
59+
- Self-optimizing system
60+
- Confidence scores guide decision-making
61+
62+
### Negative
63+
64+
- Requires historical data to be effective
65+
- Cold start problem for new task types
66+
- Pattern storage overhead
67+
- Potential overfitting to specific scenarios
68+
69+
## Implementation
70+
71+
- **Pattern Storage**: `coordination/knowledge-base/learned-patterns/patterns-latest.json`
72+
- **Learning Script**: `llm-mesh/moe-learning/evaluators/pattern-learner.sh`
73+
- **Metrics**: `coordination/metrics/learning/learner-metrics.jsonl`
74+
75+
## Metrics
76+
77+
Target metrics after 1000 routing decisions:
78+
79+
| Metric | Target | Current |
80+
|--------|--------|---------|
81+
| Routing Accuracy | 90% | 85% |
82+
| Avg Confidence | 0.80 | 0.75 |
83+
| Pattern Coverage | 95% | 90% |
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# ADR 003: Worker Pool Management Strategy
2+
3+
**Status**: Accepted
4+
**Date**: 2025-11-20
5+
6+
## Context
7+
8+
Workers are spawned on-demand, causing:
9+
- 2-3 second cold start delay
10+
- Inefficient resource usage
11+
- Slow task execution
12+
13+
## Decision
14+
15+
Implement pre-warmed worker pool with:
16+
17+
- **Min Pool Size**: 5 workers
18+
- **Max Pool Size**: 20 workers
19+
- **Scale Up**: When queue > 10 tasks
20+
- **Scale Down**: When idle > 5 minutes
21+
22+
## Implementation
23+
24+
```javascript
25+
class WorkerPool {
26+
constructor() {
27+
this.minSize = 5;
28+
this.maxSize = 20;
29+
this.pool = [];
30+
this.initializePool();
31+
}
32+
33+
initializePool() {
34+
for (let i = 0; i < this.minSize; i++) {
35+
this.spawnWorker();
36+
}
37+
}
38+
39+
scaleUp() {
40+
if (this.pool.length < this.maxSize) {
41+
this.spawnWorker();
42+
}
43+
}
44+
45+
scaleDown() {
46+
if (this.pool.length > this.minSize) {
47+
this.killIdleWorker();
48+
}
49+
}
50+
}
51+
```
52+
53+
## Results
54+
55+
- Cold start: 2000ms → 100ms (20x faster)
56+
- Resource utilization: 20% → 75%
57+
- Task throughput: 10/min → 50/min

0 commit comments

Comments
 (0)