docs/rtmp-integration-plan.md

RTMP Integration Plan for Stack 2.9

Objective: Replace the legacy MCP‑based tool layer with a native Python‑centric tool system while preserving existing functionality and enabling easier extension.

---

1. Architecture Overview (text diagram)

+-------------------+          +--------------------+          +-------------------+
|  Stack 2.9 Core   |  <----> |  Tool Registry     |  <----> |  Python Tool Impl |
| (models, prompts) |          | (register/unregister) |      | (functions)      |
+-------------------+          +--------------------+          +-------------------+
         ^                                 ^                       ^
         |                                 |                       |
         |                                 |                       |
         |            +--------------------+-----------------------+
         |            |  Tool Invocation Layer (async)          |
         |            |  - validates input (zod -> pydantic)   |
         |            |  - checks permissions                  |
         |            |  - streams progress & result            |
         |            +----------------------------------------+
         |
         v
+-------------------+          +--------------------+          +-------------------+
|   MCP Bridge      |  (legacy) |  MCP Integration   |  (keep for backward) |
+-------------------+          +--------------------+          +-------------------+

The new system runs entirely in Python, mirroring the API surface of the existing MCPTool class but using native classes.

---

2. Implementation Phases

Phase 0 – Prep & Tool Discovery

1. Add a tools/ package under stack-2.9/src/ (e.g. src/tools).
2. Create a base Tool abstract class defining:
   • name, description, input_schema (pydantic), output_schema.
   • call(self, **params) -> Any (may be async).
   • Optional hooks: validate, check_permissions, is_destructive, summary.
3. Add a tool_registry.py singleton with:
   • register(tool: Tool)
   • unregister(name: str)
   • get(name: str) -> Tool
   • list() -> List[Tool]
4. Implement auto‑loader that scans src/tools/** for subclasses of Tool and registers them at import time (mirrors RTMP's loadAgentsDir).
5. Write unit tests for registry behavior.

Phase 1 – Core Tool System

1. Define schema handling using pydantic (lightweight, validates JSON schemas). Provide a helper zod_to_pydantic if needed for future port‑over.
2. Implement ToolInvoker class:
   • async invoke(name, params, context)
   • Performs input validation, permission check, runs tool.call().
   • Streams progress via async generator (yield progress).
3. Add permission framework similar to RTMP's checkPermissions:
   • Global policy file tools/permissions.yaml (allow/deny patterns).
   • PermissionChecker reads the policy, supports always_allow, always_deny, ask_user.
4. Create example native tools mirroring MCP built‑ins:
   • ReadFileTool, WriteFileTool, WebSearchTool, RunCommandTool, GitTool.
   • Each lives in its own module under src/tools/.
5. Update Stack entrypoint (stack-2.9/stack.py or CLI) to import the registry and expose the new tool list to the LLM prompt (replace MCP list_tools).

Phase 2 – Migration Layer (MCP Compatibility)

1. Keep the existing MCPIntegration class but make it a thin wrapper around the new registry:
   • On register_tool, forward to tool_registry.register.
   • On call_tool, forward to ToolInvoker.invoke.
2. Add adapter classes for each legacy MCP tool that simply subclass the new Tool and delegate to the original implementation (e.g., LegacyReadFileAdapter). This ensures any existing agent definitions continue to work.
3. Write migration script scripts/migrate_mcp_tools.py that:
   • Loads all legacy MCP tools (from RTMP/tools).
   • Generates stub Python classes in src/tools/ with matching signatures.
   • Updates enhancements/collaboration/mcp_integration.py to use the new registry.
4. Run integration tests to confirm both legacy and native tools are discoverable.

Phase 3 – Full Cut‑over & Cleanup

1. Update prompt generation (agentToolUtils etc.) to pull tool definitions from the Python registry instead of MCP metadata.
2. Deprecate the MCP client in collaboration/mcp_integration.py:
   • Mark as deprecated with warning logs.
   • Remove automatic auto‑registration of MCP tools.
3. Remove the RTMP/tools directory from the repo (or archive it under legacy/).
4. Extend documentation and add migration guide for developers.
5. bump version to 2.9.1.

---

3. File Structure for New Tool System

stack-2.9/
├─ src/
│  ├─ tools/
│  │  ├─ __init__.py               # registers all tools on import
│  │  ├─ base.py                    # abstract Tool class
│  │  ├─ registry.py                # singleton registry implementation
│  │  ├─ invoker.py                 # ToolInvoker handling validation & permissions
│  │  ├─ permissions.yaml           # policy file (allow/deny patterns)
│  │  ├─ read_file.py               # native ReadFileTool
│  │  ├─ write_file.py              # native WriteFileTool
│  │  ├─ web_search.py              # native WebSearchTool (uses web_search tool)
│  │  ├─ run_command.py            # native RunCommandTool (subprocess)
│  │  └─ git_tool.py                # native GitTool (uses GitPython or subprocess)
│  └─ enhancements/
│     └─ collaboration/
│        ├─ __init__.py
│        ├─ mcp_integration.py      # thin wrapper -> registry
│        └─ conversation_state.py
├─ docs/
│  └─ rtmp-integration-plan.md      # <-- this file
└─ scripts/
   └─ migrate_mcp_tools.py          # migration helper

---

4. Migration Path

Step	Action	Outcome
1️⃣	Add native registry & base classes	Core infrastructure ready
2️⃣	Implement native equivalents of all built‑in MCP tools	Feature parity
3️⃣	Wrap legacy tools with adapters & register them via the new registry	Zero‑breakage for existing agents
4️⃣	Switch list_tools calls to registry.list()	Prompt now reflects native tools
5️⃣	Deprecate MCP client, remove old RTMP/tools folder	Clean codebase
6️⃣	Update documentation & version bump	Done

---

5. Security & Permissions

• Schema validation via pydantic ensures only expected fields reach the implementation.
• PermissionChecker reads tools/permissions.yaml:

  always_allow:
    - read_file
    - web_search
  always_deny:
    - run_command: "*"
  ask_user:
    - write_file
    - git_operation
  

• Hooks can raise PermissionDenied which bubbles up to the LLM as a tool‑use error message.
• All destructive tools (write_file, git_operation, run_command with sudo) must be marked is_destructive=True to trigger explicit user confirmation.

---

6. Documentation & Testing

• Add docstrings to each tool class (auto‑generated API reference via pydoc).
• Write pytest suites covering:
  • Registry CRUD
  • Input validation failures
  • Permission matrix (allow/deny/ask)
  • End‑to‑end invocation through ToolInvoker.
• Update README.md with a “Native Tool System” section describing how to add new tools.

---

7. Risks & Mitigations

Risk	Impact	Mitigation
Regression of existing agents relying on MCP metadata format	Breaks user workflows	Keep adapter layer until next major release; run full suite of agent integration tests.
Permission policy mis‑configuration causing silent denial	User confusion	Fail fast with clear error messages; add a startup sanity check that all tools have a permission rule.
Performance regression due to Python validation overhead	Slower turn latency	Cache compiled pydantic models; benchmark against MCP’s JSON‑schema validation (expected to be comparable).

---

8. Next Steps

1. Commit the skeleton (src/tools/ with base classes & registry).
2. Implement ReadFileTool and WriteFileTool as proof‑of‑concept.
3. Run the migration script to generate adapters for the remaining MCP tools.
4. Update the Stack entrypoint to expose the new tool list.
5. Iterate on tests and documentation.

---

Prepared by the sub‑agent tasked with RTMP integration.