---
description: How to deploy the AI Hub application to the production server (192.168.68.113)
---

This workflow automates the deployment of the Cortex Hub to the production server located at `192.168.68.113`. 
The production server runs the application out of `/home/coder/project/cortex-hub`.

**MAIN KNOWLEDGE POINT:** Agents and Users should refer to `.agent/workflows/deployment_reference.md` to understand the full proxy and architecture layout prior to running production debugging.

---

## ⚠️ MANDATORY: Run verification BEFORE committing / deploying

### 1. Sanity Check Backend Startup
Before committing or deploying, ensure the backend can start without immediate crashes (e.g., NameErrors or import issues):

```bash
cd /app/ai-hub && export DEVELOPER_MODE=true && export DATABASE_URL="sqlite:///./test_cortex.db" && uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-level debug
```
Ensure the logs show `Started server process [...]` and `Waiting for application startup.` without errors.

### 2. Run file sync tests
Any change to the following files **requires running the file sync integration tests first** and all must pass before pushing:

- `agent-node/src/agent_node/core/watcher.py`
- `agent-node/src/agent_node/core/sync.py`
- `agent-node/src/agent_node/node.py`
- `ai-hub/app/core/grpc/services/assistant.py`
- `ai-hub/app/core/grpc/services/grpc_server.py`
- `ai-hub/app/core/grpc/core/mirror.py`
- `ai-hub/app/protos/agent.proto`

**Run small file tests (Cases 1–4 and 9, ~5s) — the minimum bar before any commit:**

// turbo
```bash
source /app/.agent/utils/env_loader.sh
sshpass -p "$REMOTE_PASSWORD" ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} \
  "echo '$REMOTE_PASSWORD' | sudo -S docker exec \
    -e SYNC_TEST_BASE_URL=http://127.0.0.1:8000 \
    -e SYNC_TEST_USER_ID=9a333ccd-9c3f-432f-a030-7b1e1284a436 \
    ai_hub_service \
    python3 -m pytest /tmp/test_file_sync.py::TestSmallFileSync -v -s -p no:warnings 2>&1"
```

> The test file is deployed automatically by `remote_deploy.sh`. If you need to refresh it without a full deploy, see `/file_sync_tests` workflow.

All 5 small-file cases must show `PASSED`. If any fail, **do not deploy** — diagnose the regression first (see `/file_sync_tests` for troubleshooting).

**Run macOS File Persistence tests (~30s) — ensures Mac Mini symlink bugs didn't regress:**

// turbo
```bash
awk '/```bash/{flag=1; next} /```/{flag=0} flag' .agent/workflows/mac_mini_file_sync_persistence.md > /tmp/mac_test.sh && bash /tmp/mac_test.sh
```
Or view the `.agent/workflows/mac_mini_file_sync_persistence.md` workflow for the verbose script. If the file output is deleted or missing after 20 seconds, **do not deploy**.

---

## Deployment Steps

1. **Automated Secret Fetching**: The `scripts/remote_deploy.sh` script will automatically pull the production password from the GitBucket Secret Vault if the `GITBUCKET_TOKEN` is available in `/app/.env.gitbucket`.
2. **Environment Configuration**: Because the application defaults to an open/local configuration (Day 0 loopback mode) without OIDC, it is critical that the production `.env` file is populated with the correct authenticating variables if using SSO:
   - `OIDC_CLIENT_ID=cortex-server`
   - `OIDC_CLIENT_SECRET=change-me-at-runtime`
   - `OIDC_SERVER_URL=http://localhost:8080`
   - `OIDC_REDIRECT_URI=http://localhost:8002/api/v1/users/login/callback`
   *(Ensure these are mapped in your production deployment toolchain or manually placed on the server under `/home/coder/project/cortex-hub/.env`)*
3. **Auto-Generate API Docs**: Before deploying, if any backend API changes have been made, regenerate the documentation:
   ```bash
   cd /app/ai-hub && python3 /app/.agent/utils/generate_api_docs.py
   ```
4. **Sync**: Sync local codebase to `/tmp/cortex-hub/` on the server.
5. **Proto & Version Management**: 
   - If `ai-hub/app/protos/agent.proto` has changed, regenerate stubs:
     ```bash
     cd /app/ai-hub/app/protos && python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. agent.proto
     cd /app/agent-node && python3 -m grpc_tools.protoc -Iprotos --python_out=. --grpc_python_out=. protos/agent.proto
     ```
   - **Triggering Client Updates**: To force all active agents (like the Mac Mini) to pull new code or skills, bump the version in `/app/agent-node/VERSION`.
   - **Auto-Update**: Once deployed, agents check for version mismatches every 5 minutes and will automatically re-bootstrap themselves if a newer version is detected on the Hub.
6. **Migrate & Rebuild**: Overwrite production files and run `bash scripts/local_rebuild.sh` on the server. Use `--fast` to skip container rebuilds for pure code/Python updates.
7. **Post-Deployment Health Check**: Perform a backend connectivity check (Python Trick). Only use `/frontend_tester` as a last resort if UI behavior is suspect.

### Automated Command
```bash
# This script handles authentication, syncing, and remote rebuilding.
bash /app/scripts/remote_deploy.sh

# If you want to deploy quickly without rebuilding the standalone agent binaries:
bash /app/scripts/remote_deploy.sh --skip-binaries

# ULTRA-FAST: Skip container rebuilds AND binary builds (best for code-only tweaks):
bash /app/scripts/remote_deploy.sh --fast
```

---

## ⚠️ MANDATORY: Post-Deployment Verification

After deploy completes, run **both** checks:

### 1. Basic API health check
```bash
source /app/.agent/utils/env_loader.sh
sshpass -p "$REMOTE_PASSWORD" ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} \
  "echo '$REMOTE_PASSWORD' | sudo -S docker exec ai_hub_service python3 -c \"
import requests
headers = {'X-User-ID': '9a333ccd-9c3f-432f-a030-7b1e1284a436'}
r = requests.get('http://localhost:8000/api/v1/nodes/test-node-1/status', headers=headers)
print(f'Status: {r.status_code}')
print(r.text[:200])
\""
```
**Expected Outcome**: `200 OK` and `{"status": "online", ...}`.

### 2. File sync regression test (re-run after deploy)

// turbo
```bash
source /app/.agent/utils/env_loader.sh
sshpass -p "$REMOTE_PASSWORD" ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} \
  "echo '$REMOTE_PASSWORD' | sudo -S docker exec \
    -e SYNC_TEST_BASE_URL=http://127.0.0.1:8000 \
    -e SYNC_TEST_USER_ID=9a333ccd-9c3f-432f-a030-7b1e1284a436 \
    ai_hub_service \
    python3 -m pytest /tmp/test_file_sync.py::TestSmallFileSync -v -p no:warnings 2>&1"
```
All 5 must pass. If any fail post-deploy, **rollback immediately** using `git revert` and re-deploy.

### Visual Verification (Worst Case)
If the backend check passes but UI behaviors are suspect or need deeper frontend validation, use the `/frontend_tester` workflow:
```bash
/frontend_tester
```
