Skill File-System Refactor Implementation Plan

Overview

This document outlines the transition fully away from database-driven skill storage towards a pure File-System-Based Architecture. Skills will become directories on the server grouped by features (e.g., swarm_control, voice_chat), simplifying the user interface and maximizing flexibility (enabling git versioning, native file inclusion, etc.).

Phase 1: Backend Architecture & Tool Loader Reform

1.1 Data Structure Updates

Define a base directory configuration settings.SKILLS_DIR (e.g., /app/data/skills).
Deprecate existing SQLAlchemy models for Skill and SkillFile (do not delete db tables yet for backup).

1.2 Hot-Reload Engine & File Watcher (`app.core.skills.loader`)

Create a FileSystemSkillLoader script.
The loader will os.walk() through settings.SKILLS_DIR on boot, or use a background File System Watcher (e.g., watchdog) to dynamically update capabilities.
- Level 1 directories define the "Feature" (e.g., swarm_control, voice_chat).
- Level 2 directories define the "Skill Unique ID" (e.g., get_weather).
Reads SKILL.md from these folders to mount Markdown schemas and execute scripts.

1.3 Invisible RBAC Metadata (`.metadata.json`)

If a skill folder is newly created (either manually on the server or via API) or if pre-existing files are detected, the loader will check for a hidden .metadata.json file.
Server Boot Initialization: When the server boots and recursively scans the directory structure, it will auto-generate .metadata.json for any existing folder that lacks one, assigning ownership defaults (e.g., admin).
Backend API Creation: If the user creates the skill from the frontend UI, the backend implicitly writes a .metadata.json assigning owner_id to that user.
Manual Server Manipulation: If an administrator copies a folder directly to the server via terminal, the File Watcher detects the missing .metadata.json and auto-generates it, defaulting the owner_id to the cluster admin.
The frontend UI will explicitly hide .metadata.json from the Visual Explorer so developers aren't bothered by system files. Permission modifications in the future will interact exclusively with this file.

1.4 Update `tool.py`

Modify _execute_system_skill and get_tools inside app/core/services/tool.py to route through the new FileSystemSkillLoader cache instead of invoking the database sessions. The .metadata.json overrides will dictate authorization blocks.

Phase 2: Refactoring the REST API (`app.api.routes.skills`)

2.1 Decouple from SQLAlchemy

Rewrite the GET /api/v1/skills/ endpoint to return JSON payloads mapping the physical directories (e.g., reading directories, returning the tree object).
Replace POST, PUT, DELETE operations with standard Python os and shutil commands:
- Creating a skill creates a folder.
- Adding a file creates a physical file inside that folder.
- Deleting a skill calls shutil.rmtree().
Expose an endpoint to recursively read the tree structure of a skill (/api/v1/skills/{feature}/{skill_id}/tree).

Phase 3: Frontend Simplification Experience (`ai_unified_frontend`)

3.1 Strip Declarative Database Forms

Eliminate modal logic dealing with name, description, skill_type, and is_system checkboxes from SkillsPage.js.
Provide a pure UI File Explorer that does not use forms to "generate" a skill. Instead, users simply right-click to "New Folder", typing the exact directory name they want (e.g., get_weather).
There is no backend conversion or "slugifying" of names. If a user types illegal characters for a folder name, the underlying OS rejection is simply passed back to the UI. The human-readable title and emojis will be parsed directly from the SKILL.md content (e.g., ### Skill Name: Get Weather 🌤️).

3.2 Direct VSCode-Like IDE Integration

Turn the SkillsPage layout into a two-panel IDE layout:
- Left Sidebar: File Explorer detailing the tree (Feature > Folder Name > Files).
- Right Panel: A large Monaco Editor or CodeMirror instance loading the physical file content.
Support creation of arbitrary code assets (Python, textual configs) alongside the SKILL.md via UI "Right Click -> New File".

Considerations

Migration Script: Before deleting DB dependencies, we optionally need a Python script to sweep existing skills out of PostgreSQL and physically write them into /app/data/skills/.
Access Security: If multi-tenancy is still important, settings.SKILLS_DIR could be parameterized per user (/app/data/skills/{user_id}/...) or we accept skills as global components for the Hub.
Execution Sandbox: Existing nodes and bash execution sandboxes will remain perfectly intact. The only change is how the definitions arrive at the orchestrator.