Fork: 0
yangyangxie / cortex-hub
Transfer to URL with SHA Find file
Newer
Older
cortex-hub / .agent / workflows / deployment_reference.md
@Antigravity AI Antigravity AI 13 hours ago 9 KB Cleanup skill logic files and update deployment scripts
Raw Blame History

description: Reference Guide for Cortex Hub Production Architecture, proxy networking, and debugging deployments.

Cortex Hub: Deployment & CI/CD Architecture Reference

This document serves as the comprehensive reference guide for the deployment architecture of the Cortex Hub AI system. It maps the journey of a request from the external internet to the backend code, details the containerized architecture, and outlines the automated deployment (CI/CD) paths, to help debug future production issues.

1. High-Level Architecture Flow

When an external user visits https://ai.jerxie.com, the request cascades through the following components:

flowchart TD
    A[External User] -->|HTTPS :443| B(Envoy Control Plane Proxy\nHost IP: 192.168.68.90 :10001)
    B -->|HTTP :80| C{Nginx Gateway\nContainer: ai_unified_frontend\nHost IP: 192.168.68.113 :8002}
    
    C -->|Static Files / JS / CSS| D[React Frontend Build]
    C -->|/api/v1/* Routing| E(FastAPI Backend\nContainer: ai_hub_service\nExposed: :8000 internally)

Components Breakdown

1. Envoy Proxy (192.168.68.90):

  • Role: Primary Edge Proxy. Terminates external SSL (HTTPS), handles external SNI routing (matching ai.jerxie.com), and passes decrypted traffic internally.
  • Control API: Envoy configurations are manipulated dynamically at http://192.168.68.90:8090/.
  • Important Note: When Envoy forwards traffic internally, the scheme changes from HTTPS to HTTP.

2. Nginx Gateway (192.168.68.113 - ai_unified_frontend container):

  • Role: Application Gateway. Handles the static React frontend natively, and reverses proxies all /api/v1/ calls directly to the Python backend.
  • Network Binding: Bound to host port 8002 (internal container port 80) to avoid port collisions with other host services.
  • Protocol Map: Converts connection headers between "upgrade" (WebSockets) and "close" (Standard HTTP), and passes Envoy's original scheme (X-Forwarded-Proto) along so FastAPI knows it originated as HTTPS.

3. Uvicorn / FastAPI Backend (ai_hub_service container):

  • Role: Core application logic. Needs to be aware it is sitting behind proxies to compute accurate callback URLs for tasks like OIDC Login, which inherently depend on accurate external domains and schemas.
  • Configuration: Runs with --proxy-headers and --forwarded-allow-ips="*" to ensure it trusts X-Forwarded-Proto variables injected by Nginx.

2. CI/CD Pipeline & Deployment Scripts

The code moves from development to production through a formalized sequence, managed by dedicated local shell scripts.

The Scripts

  1. scripts/remote_deploy.sh (The Trigger)

    • Where it runs: Locally on your dev machine
    • What it does:
      1. Uses rsync over SSH (sshpass) to securely copy local workspace (/app/) changes onto the production server 192.168.68.113 under a temporary /tmp/ directory.
      2. It specifically excludes massive or unnecessary folders (.git, node_modules, __pycache__).
      3. Overwrites the destination project folder (/home/coder/project/cortex-hub) taking care to retain system permissions.
      4. SSH triggers the scripts/local_rebuild.sh script centrally on the production server.

  2. scripts/local_rebuild.sh (The Builder)

    • Where it runs: Server 192.168.68.113
    • What it does:
      1. Detects the deployment environment by checking for override files in deployment/.
      2. Destroys the old running containers.
      3. Triggers Docker Compose with multiple layers:
        • -f docker-compose.yml (Generic common config)
        • -f deployment/jerxie-prod/docker-compose.production.yml (Jerxie-specific overrides)
      4. Rebuilds and starts the containers.
      5. Performs automated database migrations running parallel idempotent logic via the Uvicorn startup lifecycle.

How to Release

You or an Agent must safely pass the authentication key into the script from the command line:

REMOTE_PASS='<PASSWORD>' bash scripts/remote_deploy.sh

3. Decoupled Folder Structure

To maintain a clean repository and allow for generic onboarding while keeping Jerxie's specific production requirements separate, the following layout is enforced:

  • docker-compose.yml: Baseline configuration. Uses local Docker volumes and generic localhost endpoints. Default for new users.
  • deployment/jerxie-prod/: Contains Jerry's specific production overrides (NFS volume on 192.168.68.90, SSL/OIDC endpoints for ai.jerxie.com).
  • scripts/: Centralized automation scripts for remote sync, local rebuilding, and node registration. For any automated AI attempting to debug or push changes:
  • Primary Source of Truth: This file (.agent/workflows/deployment_reference.md) defines the architecture rules.
  • Subflow - Deployment: Look at .agent/workflows/deploy_to_production.md for explicit directory movement maps and scripting.
  • Subflow - Envoy Control Map: Refer to .agent/workflows/envoy_control_plane.md if the backend successfully mounts on the remote machine but ai.jerxie.com/api/ throws 404s/503s.

5. Agent Node Auto-Update Architecture

Cortex supports zero-touch maintenance for distributed agent nodes (like Mac Minis or remote VMs).

How it Works

  1. Version Monitoring: The running Agent process (agent_node/main.py) runs a background AutoUpdater thread.
  2. Polling: Every 300 seconds, the agent calls GET /api/v1/agent/version on the Hub using the SECRET_KEY for authentication.
  3. Trigger: If the Hub reports a version higher than the one found in the local VERSION file, the agent:
    • Downloads the latest code bundle (including skills/) from GET /api/v1/agent/download.
    • Delegates to bootstrap_installer.py --update-only to extract and install new dependencies.
    • Restarts itself (os.execv) to load the new code.
  4. Resiliency: The installer uses --ignore-installed to prevent common Python/Anaconda metadata errors from stalling updates.

4. Reference Troubleshooting & Known Pitfalls

If production encounters a bug or routing fails, these are the historically primary offenders:

1. OIDC Redirection Errors (Auth Flips to HTTP)

Symptoms: OAuth login fails because Dex/Auth redirects the user back to http://ai.jerxie.com/api/... instead of https://. Root Cause: FastAPI believes it's serving HTTP because Nginx didn't forward the Envoy proxy scheme. Verification Check:

  • Check app/ai-hub/Dockerfile: Ensure uvicorn terminates with --proxy-headers --forwarded-allow-ips "*".
  • Check nginx.conf: Ensure proxy_set_header X-Forwarded-Proto relies on the HTTP dynamically captured from Envoy, NOT a hard-coded $scheme string.

2. WebSocket Hanging / HTTP Request Failing

Symptoms: Normal API calls return strange transport errors, or WebSocket voice channels refuse to upgrade. Root Cause: Misconfigured HTTP Upgrade logic. Setting Connection "upgrade" unconditionally on an Nginx location breaks normal HTTP REST calls. Verification Check:

  • Check nginx.conf mappings. It must have: 
    map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
    ... and passed safely into the proxy_set_header Connection $connection_upgrade; field.

3. "Port is already allocated" (Container Fails to Deploy)

Symptoms: Backend loads normally, but Frontend acts as dead or Exit 1 silently during deploy_remote.sh. Root Cause: Binding collisions on the production host 192.168.68.113. (e.g., trying to bind Nginx to host port 8000 when another container python3_11 uses it). Verification Check:

  • Run docker ps -a on 192.168.68.113.
  • Ensure the frontend port (8002:80) mapped inside docker-compose.yml does not overlap with any live container binding arrays.

4. Envoy Cluster Sync Outages

Symptoms: curl -v https://ai.jerxie.com dumps a generic 404 or 503 Service Unavailable with server: envoy. Root Cause: The Envoy FilterChain (Listener SNI Map) doesn't trace back to a correct, valid Docker IP:Port allocation. Verification Check:

  • Make sure portValue in the JSON Endpoint equates to the one published in docker-compose.yml (8002 vs 8000). If mismatched, you must format a JSON package and POST it to /add-cluster utilizing the EnvoryControlPlane workflow.

5. OIDC Login Failures (Stale/Empty Client ID)

Symptoms: User attempts to login but is immediately rejected by Auth. Jerxie (Dex). Dex logs show: level=ERROR msg="failed to parse authorization request" err="Invalid client_id (\"\")." Root Cause: API routes were incorrectly capturing OIDC settings from settings at the module's initial import time rather than at request time. Empty/Default environment variables were being permanentized until a full container restart. Verification Check:

  • Check ai_hub_service logs for: Initiating OIDC login. Client ID: 'cortex-server'.
  • Ensure that the settings singleton is accessed inside the route function (or via a helper like get_oidc_urls) to ensure dynamic resolution.

6. Periodic Hub "Freezes" or Disconnections

Symptoms: The ai.jerxie.com UI becomes unresponsive (spinning loaders or time-outs) for 1-2 minutes before suddenly recovering. No backend crashes are recorded. Root Cause: SQLite on NFS. The production deployment uses an NFS-mounted volume (192.168.68.90) for the relational database. SQLite's write-ahead-logging (WAL) and intense locking requirements are highly incompatible with network filesystem latencies. Any minor network spike or high NFS load causes the Hub process to block entirely while waiting for a lock. Resolution:

  • High Priority: Move ai_hub.db to a Local Volume or a Local Host Path on 192.168.68.113. NFS should be reserved for static sync folders or bulky assets, never for the primary relational database.