Cortex Hub: Deployment & CI/CD Architecture Reference

This document serves as the comprehensive reference guide for the deployment architecture of the Cortex Hub AI system. It maps the journey of a request from the external internet to the backend code, details the containerized architecture, and outlines the automated deployment (CI/CD) paths, to help debug future production issues.

1. High-Level Architecture Flow

When an external user visits https://ai.jerxie.com, the request cascades through the following components:

flowchart TD
    A[External User] -->|HTTPS :443| B(Envoy Control Plane Proxy\nHost IP: 192.168.68.90 :10001)
    B -->|HTTP :80| C{Nginx Gateway\nContainer: ai_unified_frontend\nHost IP: 192.168.68.113 :8002}
    
    C -->|Static Files / JS / CSS| D[React Frontend Build]
    C -->|/api/v1/* Routing| E(FastAPI Backend\nContainer: ai_hub_service\nExposed: :8000 internally)

Components Breakdown

1. Envoy Proxy (192.168.68.90):

• Role: Primary Edge Proxy. Terminates external SSL (HTTPS), handles external SNI routing (matching ai.jerxie.com), and passes decrypted traffic internally.
• Control API: Envoy configurations are manipulated dynamically at http://192.168.68.90:8090/.
• Important Note: When Envoy forwards traffic internally, the scheme changes from HTTPS to HTTP.

2. Nginx Gateway (192.168.68.113 - ai_unified_frontend container):

• Role: Application Gateway. Handles the static React frontend natively, and reverses proxies all /api/v1/ calls directly to the Python backend.
• Network Binding: Bound to host port 8002 (internal container port 80) to avoid port collisions with other host services.
• Protocol Map: Converts connection headers between "upgrade" (WebSockets) and "close" (Standard HTTP), and passes Envoy's original scheme (X-Forwarded-Proto) along so FastAPI knows it originated as HTTPS.

3. Uvicorn / FastAPI Backend (ai_hub_service container):

• Role: Core application logic. Needs to be aware it is sitting behind proxies to compute accurate callback URLs for tasks like OIDC Login, which inherently depend on accurate external domains and schemas.
• Configuration: Runs with --proxy-headers and --forwarded-allow-ips="*" to ensure it trusts X-Forwarded-Proto variables injected by Nginx.

2. CI/CD Pipeline & Deployment Scripts

The code moves from development to production through a formalized sequence, managed by dedicated local shell scripts.

The Scripts

1. remote_deploy.sh (The Triggger)

   • Where it runs: Locally on your dev machine
   • What it does:
     1. Uses rsync over SSH (sshpass) to securely copy local workspace (/app/) changes onto the production server 192.168.68.113 under a temporary /tmp/ directory.
     2. It specifically excludes massive or unnecessary folders (.git, node_modules, __pycache__).
     3. Overwrites the destination project folder (/home/coder/project/cortex-hub) taking care to retain system permissions.
     4. SSH triggers the local_deployment.sh script centrally on the production server.

2. local_deployment.sh (The Builder)

   • Where it runs: Server 192.168.68.113
   • What it does:
     1. Destroys the old running containers.
     2. Triggers Docker Compose (docker compose up -d --build --remove-orphans) to rebuild the application context and discard deprecated container setups (e.g., when the UI shifted into Nginx).
     3. Performs automated database migrations running parallel idempotent logic (app/db/migrate.py) via the Uvicorn startup lifecycle.

How to Release

You just run:

bash /app/remote_deploy.sh

3. Reference Troubleshooting & Known Pitfalls

If production encounters a bug or routing fails, these are the historically primary offenders:

1. OIDC Redirection Errors (Auth Flips to HTTP)

Symptoms: OAuth login fails because Dex/Auth redirects the user back to http://ai.jerxie.com/api/... instead of https://. Root Cause: FastAPI believes it's serving HTTP because Nginx didn't forward the Envoy proxy scheme. Verification Check:

• Check app/ai-hub/Dockerfile: Ensure uvicorn terminates with --proxy-headers --forwarded-allow-ips "*".
• Check nginx.conf: Ensure proxy_set_header X-Forwarded-Proto relies on the HTTP dynamically captured from Envoy, NOT a hard-coded $scheme string.

2. WebSocket Hanging / HTTP Request Failing

Symptoms: Normal API calls return strange transport errors, or WebSocket voice channels refuse to upgrade. Root Cause: Misconfigured HTTP Upgrade logic. Setting Connection "upgrade" unconditionally on an Nginx location breaks normal HTTP REST calls. Verification Check:

• Check nginx.conf mappings. It must have:

  map $http_upgrade $connection_upgrade {
      default upgrade;
      ''      close;
  }

  ... and passed safely into the proxy_set_header Connection $connection_upgrade; field.

3. "Port is already allocated" (Container Fails to Deploy)

Symptoms: Backend loads normally, but Frontend acts as dead or Exit 1 silently during remote_deploy.sh. Root Cause: Binding collisions on the production host 192.168.68.113. (e.g., trying to bind Nginx to host port 8000 when another container python3_11 uses it). Verification Check:

• Run docker ps -a on 192.168.68.113.
• Ensure the frontend port (8002:80) mapped inside docker-compose.yml does not overlap with any live container binding arrays.

4. Envoy Cluster Sync Outages

Symptoms: curl -v https://ai.jerxie.com dumps a generic 404 or 503 Service Unavailable with server: envoy. Root Cause: The Envoy FilterChain (Listener SNI Map) doesn't trace back to a correct, valid Docker IP:Port allocation. Verification Check:

• Query the Control Plane API: curl -s http://192.168.68.90:8090/get-cluster?name=_ai_unified_server.
• Make sure portValue in the JSON Endpoint equates to the one published in docker-compose.yml (8002 vs 8000). If mismatched, you must format a JSON package and POST it to /add-cluster utilizing the EnvoryControlPlane workflow.