---
description: Troubleshooting Guide for File Sync and Node Issues
---

# Troubleshooting Guide: Sync & Node Debugging

This workflow documents common issues encountered with the AI Hub file synchronization and node management, along with strategies and scripts used to diagnose and resolve them.

## 1. File Synchronization Dropping Large Files
**Symptoms**:
- A node successfully creates a large file (e.g., using `dd` or `cp`).
- The file does not appear on the central AI Hub mirror.
- Other connected nodes do not receive the file chunk broadcasts.

**Troubleshooting Steps & Checklist**:
1. Check Proto3 compliance: Ensure you are directly accessing `payload.offset` rather than using `payload.HasField("offset")` for primitive variables, as Proto3 does not reliably support `.HasField` for int64.
2. Verify Temporary File filtering: Ensure the workspace `.watcher` and Server `.mirror` are actively ignoring files ending with `.cortex_tmp` and `.cortex_lock` to prevent recursion loops (Sync Echo).
3. Validate "Empty Workspace" Initialization: When creating `source="empty"` workspaces, verify that the AI Hub actually sends a `START_WATCHING` gRPC signal to the target nodes. Without this signal, nodes will create the empty directory but the `watchdog` daemon will not attach to it, resulting in zero outbound file syncs.
4. Ensure target node is properly receiving "on_closed" events for larger files. Writing a 10MB file via `dd` triggers 60+ `on_modified` events, which can occasionally choke the stream. Implementing an `on_closed` forced-sync is more reliable.
## 2. macOS Ghost Deletion Loop (Infinite Synchronized File Removals)
**Symptoms**:
- A file created natively on a macOS node (via `echo`, UI, or `cp`) syncs upward perfectly, but deletes itself globally a few seconds later.
- Reading the agent logs shows non-stop back-to-back entries mapping: `[📁⚠️] Watcher EMITTING DELETE to SERVER` and `[📁🗑️] Delete Fragment`. 
- Atomic network uploads repeatedly cycle identically over and over indefinitely.

**Troubleshooting Steps & Checklist**:
1. Validate Zero-byte payload support: macOS frequently produces two immediate events (Creation: 0 bytes and Modification: 100 bytes). If the watcher chunk reader prematurely breaks on `if not chunk` without validating `index > 0`, the Hub never learns about the file initially, causing an aggressively executed `handle_manifest` remote deletion broadcast to scrub the new file.
2. Validate `FSEvents` `os.replace` behavior: macOS's native kernel watcher queues file deletion events structurally from `os.remove`, but occasionally delivers them hundreds of milliseconds *after* a new network write has arrived utilizing the exact same path!
3. Because `FSEvents` are delayed, when the `on_deleted` event is evaluated against the `last_sync` cache, it finds the newly calculated hash block instead of the suppressed `__DELETED__` lock natively, thereby assuming the user actively erased the file by hand again.
4. Ensure `watcher.py` validates absolute disk existence explicitly before broadcasting deletions: `if os.path.exists(real_src) or os.path.lexists(event.src_path): return`. This decisively breaks the macOS recursive network loop.
## 3. Infinite Sync Echo Loop (Modification Ping-Pong)
**Symptoms**:
- A file is created or modified, and then it immediately starts "pulsing" with `.cortex_tmp` and `.cortex_lock` files appearing and disappearing infinitely.
- High CPU/Network usage on agent nodes.
- Logs show non-stop `Streaming Sync Started` and `Async File Sync Complete` for the same file.

**Troubleshooting Steps & Checklist**:
1. **Hash-Check-First Validation**: Ensure `watcher.py` calculates the file hash **before** attempting to send chunks. If the hash matches `last_sync`, the modification event was likely an "echo" from a previous network write and must be ignored.
2. **Delayed FSEvents Masking**: macOS often delivers `on_modified` events up to several seconds after a write. Even if the path was suppressed during the actual write, the event might arrive after unsuppression. A pre-emptive hash verification is the most robust defense.
3. **Filter Internal Temp Files**: Ensure all event handlers (`on_created`, `on_modified`, `on_closed`) explicitly return if the filename ends in `.cortex_tmp` or `.cortex_lock`. This prevents the watcher from trying to sync the synchronization metadata itself.

## 4. Server Root File Ownership (Jerxie Prod)
**Symptoms**:
- Agent nodes sync files to the Hub mirror directory successfully.
- The host user (`UID 1000`) is unable to manipulate or delete these files directly from the host machine because the container writes them as `root`.

**Troubleshooting Steps & Checklist**:
1. Use Python's `os.chown` in the Mirror: During the atomic swap phase on the AI Hub (`os.replace()`), forcefully capture the `os.stat()` of the parent directory.
2. Apply `os.chown(tmp_path, parent_stat.st_uid, parent_stat.st_gid)` to the `.cortex_tmp` file immediately before the final swap. This ensures the host user retains ownership of all synced data on NFS/Mounted volumes.

## 5. Ghost Nodes Attaching to New Sessions
**Symptoms**:
- An old or offline node (e.g., `synology-nas`) keeps automatically attaching itself to newly created sessions.
- Removing it from the UI or closing previous sessions does not resolve the issue.

**Troubleshooting Steps & Checklist**:
- Verify the Postgres User `preferences` schema. In the Cortex Hub, default node attachments are tied to the user profile, not just individual prior sessions.
- Run a surgical database manipulation script directly on the backend to erase the ghost node from the `default_node_ids` array.

**Example Surgical Database Script**:
```python
import sys
sys.path.append("/app")
from app.db.session import get_db_session
from app.db.models import User, Session
from sqlalchemy.orm.attributes import flag_modified

try:
    with get_db_session() as db:
        # Find the specific user and update their preferences dict
        users = db.query(User).all()
        for u in users:
            prefs = u.preferences or {}
            nodes = prefs.get("nodes", {})
            defaults = nodes.get("default_node_ids", [])
            if "synology-nas" in defaults:
                defaults.remove("synology-nas")
                nodes["default_node_ids"] = defaults
                prefs["nodes"] = nodes
                u.preferences = prefs
                flag_modified(u, "preferences")
                
        # Clean up any already corrupted sessions
        sessions = db.query(Session).filter(Session.sync_workspace_id == "YOUR_SESSION_ID").all()
        for s in sessions:
            attached = s.attached_node_ids or []
            if "synology-nas" in attached:
                attached.remove("synology-nas")
                s.attached_node_ids = attached
                flag_modified(s, "attached_node_ids")
                
        db.commit()
        print("Surgical cleanup complete.")
except Exception as e:
    print(f"Error: {e}")
```

## Useful Diagnostic SSH Commands

### Check Agent Watcher Logs
```bash
docker logs cortex-test-1 2>&1 | grep "📁👁️"
```

### Trace File Sync on the Hub
```bash
docker logs ai_hub_service | grep "dd_test_new_live.bin"
```

### Validate Container Mount Ownership
```bash
echo 'your-password' | sudo -S ls -la /var/lib/docker/volumes/cortex-hub_ai_hub_data/_data/mirrors/session-YOUR_SESSION
```

## 6. Unresponsive Terminal (Zombie PTY Processes on Local Nodes)
**Symptoms**:
- Agent node shows tasks being received in its logs.
- Executing bash commands via the AI mesh terminal hangs indefinitely or outputs nothing.
- The `ThreadPoolExecutor` or PTY bridge is gridlocked because of a previous broken shell session.

**Troubleshooting Steps & Checklist**:
1. Check the active Python process list to locate the running `agent-node/main.py`.
2. Inspect the agent stdout and stderr logs locally (e.g. `~/.cortex/agent.out.log` and `~/.cortex/agent.err.log`) for any errors or blocked threads.
3. If the process is hung on a zombie child shell, force-kill (`kill -9`) the main agent process AND all its recursive child processes (to properly clean up spawned PTY shells).
4. Restart the agent using `bash run.sh` and redirect logs appropriately.
*Note: The agent node logic (`agent-node/src/agent_node/main.py`) has been updated in `v1.0.62` to automatically harvest and kill child PTY processes when clearing orphaned instances on boot, preventing this state across restarts.*

### Useful Diagnostic Commands (Local Node)
```bash
# Locate the agent process ID
pgrep -f "agent_node/main.py"

# Kill main process and all child PTYs (Force restart)
mac_pid=$(pgrep -f "agent_node/main.py") && kill -9 $mac_pid

# Restart agent gracefully with output redirected to background logs
cd ~/.cortex/agent-node && bash run.sh > ~/.cortex/agent.out.log 2> ~/.cortex/agent.err.log &

# Tail agent errors locally
tail -n 100 ~/.cortex/agent.err.log
```

## 7. Mac Mini Specific Agent Debugging
**Symptoms**:
- Terminal hangs immediately after a large file sync (e.g., 10MB test file).
- Commands like `ls` or `cd` work once, then the PTY bridge deadlocks.
- Symptoms unique to macOS: Symlinked paths (e.g., `/tmp` -> `/private/tmp`) causing relative path mismatches.

**Troubleshooting Steps & Checklist**:
1. **Identify PTY Deadlock**: Run `lsof -p <PID> | grep ptmx`. If more than 2-3 PTYs are open during idle, or if the number of PTYs doesn't match the number of active terminal tabs, the reader thread has likely crashed.
2. **Resolve Symlink Mismatches**: macOS handles `/tmp` as a symlink. Always use `os.path.realpath()` in the `Watcher` to ensure that `relpath` calculations are consistent between the root and the modified file.
3. **Check Queue Saturation**: If the orchestrator's `task_queue` (default size 100) is full during a sync, unrelated PTY threads will block waiting for space. Bump `maxsize` to `10000` in `node.py` to handle 10MB+ sync bursts.
4. **Investigate Deadlocks with SIGQUIT**: Send `kill -3 <PID>` to the agent. This will dump all active Python thread stacks to `stdout` (viewable in `agent.out.log`), allowing you to see exactly where the TTY reader is blocked.

**Diagnostic Commands**:
```bash
# Check File Descriptors for PTY leaks on MacOS
lsof -p $(pgrep -f "agent_node/main.py") | grep ptmx

# Force a thread dump to logs for deadlock investigation
kill -3 $(pgrep -f "agent_node/main.py")

# Verify the absolute path resolution that the agent sees
/opt/anaconda3/bin/python3 -c "import os; print(os.path.realpath('/tmp/cortex-sync'))"
```

## 8. Production Health & Log Monitoring (Jerxie Prod)
A set of automation scripts are available in `.agent/utils/` to monitor the production environment at `ai.jerxie.com`.

**Prerequisites**:
- `$REMOTE_HOST`, `$REMOTE_USER`, and `$REMOTE_PASSWORD` must be set in the environment to allow SSH access to the production server.

### Available Scripts:
- `bash .agent/utils/check_prod_health.sh`: Checks AI Hub connectivity, node registration status, and internal browser service reachability.
- `bash .agent/utils/check_prod_status.sh`: Lists all running Docker containers on the production server.
- `bash .agent/utils/get_prod_logs_hub.sh`: Tails the last 200 lines of the `ai_hub_service` logs.
- `bash .agent/utils/get_prod_logs_browser.sh`: Tails the last 200 lines of the `cortex_browser_service` logs.

### When to use:
- **Node appearing 'Offline' in UI**: Run `check_prod_health.sh` to see if the Hub registration endpoint is receiving heartbeats.
- **Browser actions failing**: Run `get_prod_logs_browser.sh` to see if Playwright is encountering resource issues or container crashes.
- **Deployment verification**: After running `remote_deploy.sh`, use `check_prod_status.sh` to ensure all services are marked as `Up`.
