[WIP] Add "Human in the Loop" (HITL) Feature to CLI

[Copilot]

Thanks for assigning this issue to me. I'm starting to work on it and will keep this PR's description up to date as I form a plan and make progress.

Original issue description:

Overview

The "Human in the Loop" (HITL) feature enables the command-line interface (CLI) of First Tr(y | ai) to interactively request additional user input during model execution. This is crucial for scenarios where model output is ambiguous, requires clarification, or user approval is needed before proceeding. The goal is to improve reliability, safety, and flexibility of GenAI agent operations by involving the user at key decision points.

---

Problem Statement

Currently, CLI executions are fully automated, relying solely on model output. In certain situations, the model may need more information, clarification, or explicit user approval before continuing. Without a human-in-the-loop mechanism, these situations can lead to suboptimal results, errors, or unintended actions.

---

Objectives

• Enable CLI processes to pause and interactively query the user for additional information or confirmation.
• Allow users to provide clarifications, approve, or reject model decisions interactively.
• Ensure seamless integration with existing CLI workflows and minimal disruption for users who do not need the feature.

---

Scope

In Scope

• CLI prompt for user input when the model signals ambiguity or requests clarification.
• Mechanism for the model to flag steps as needing human intervention (e.g., via a special token or API).
• Configurable option to enable/disable HITL mode.
• Clear messaging and guidance in the CLI for user responses.
• Logging of HITL interactions for auditability.

Out of Scope

• Full GUI or web-based HITL workflows (CLI only for this phase).
• Advanced user authentication or permissions management.

---

User Stories

1. As a CLI user, I want the process to prompt me for clarification when the model is unsure or requests more input, so that I can guide the workflow to the correct outcome.
2. As a CLI user, I want the option to approve or reject actions generated by the model before they are executed.
3. As a developer, I want a simple API/mechanism for the model code to flag points that require human input, so the CLI can react accordingly.
4. As an admin, I want the ability to disable HITL for automated workflows (e.g., CI/CD or batch jobs).

---

Functional Requirements

• FR1: The CLI must detect when the model requests human input and pause execution to display a clear prompt.
• FR2: The CLI must accept and process user responses, and pass them back to the model for continued execution.
• FR3: The CLI must support confirmation prompts (yes/no) and free-form input for clarifications.
• FR4: HITL mode must be configurable (e.g., via CLI flag --hitl or config file).
• FR5: All HITL interactions must be logged with timestamps and user responses.
• FR6: If HITL is disabled and the model requests input, the CLI must handle gracefully (e.g., fail with an informative error).

---

Non-Functional Requirements

• NFR1: User prompts must be clear, concise, and contextually informative.
• NFR2: The feature should not add significant latency to CLI operations when HITL is not used.
• NFR3: Code changes must include tests for HITL logic and edge cases.

---

UX/UI Considerations

• Prompts in the CLI should clearly indicate why user input is needed.
• Provide default options (e.g., [Y/n]) where applicable.
• Allow users to abort or retry the operation gracefully.

---

Open Questions

• How should the model signal a HITL event? (e.g., special return value, exception, or API call)
• Should there be a timeout for user input?
• Should input validation be supported for certain prompts?

---

Success Metrics

• Number of successful CLI executions with HITL enabled.
• User satisfaction/feedback with HITL prompts.
• Decrease in errors due to ambiguous model output.

---

Dependencies

• Changes in the model code to support signaling HITL events.
• Updates to CLI core logic and argument parsing.

---

Milestones

1. Design and agree on the signaling mechanism for HITL events.
2. Implement CLI HITL prompting and response handling.
3. Add configuration options for enabling/disabling HITL.
4. Testing, documentation, and release.

---

Appendix

• References to similar HITL implementations in open source GenAI projects.
• Example user flows.

Fixes #18.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.