Kickoff Plan: AI Model Hub Service

This document outlines the plan for developing a central "hub" service that routes requests to various Large Language Models (LLMs) and uses PostgreSQL for metadata storage alongside FAISS for similarity search on vector data.

1. High-Level Architecture

The service will consist of three main components:

1. API Server: A web server that exposes endpoints to receive user prompts and return model responses. This will be the main entry point for all client applications.

2. LLM Router/Orchestrator: A core logic layer responsible for deciding which LLM (Gemini, DeepSeek, etc.) should handle a given request. It will also manage interactions with PostgreSQL and FAISS.

3. Vector Database (FAISS + PostgreSQL): A two-layered database system:

   • FAISS: Stores vectors (numerical representations of text). Handles high-performance similarity search.
   • PostgreSQL: Stores metadata such as conversation IDs, document titles, timestamps, and other relational data.

2. Technology Stack

• API Framework: FastAPI (Python) – High-performance, easy to learn, with automatic interactive documentation, ideal for testing and development.

• LLM Interaction: LangChain (or a similar abstraction library) – Simplifies communication with different LLM APIs by providing a unified interface.

• Vector Database:

  • FAISS: High-performance similarity search for vectors.
  • PostgreSQL: Stores metadata for vectors, such as document IDs, user data, timestamps, etc. Used for filtering, organizing, and managing relational data.

• Deployment: Docker – Containerizing the application for portability, ensuring easy deployment across any machine within the local network.

3. Development Roadmap

Phase 1: Core API and Model Integration (1-2 weeks)

• [X] Set up a basic FastAPI server.
• [X] Create a /chat endpoint that accepts user prompts.
• [X] Implement basic routing logic to forward requests to one hardcoded LLM (e.g., Gemini).
• [X] Connect to the LLM's API and return the response to the user.

Phase 2: PostgreSQL and FAISS Integration (2-3 weeks)

• [ ] Integrate PostgreSQL for metadata storage (document IDs, timestamps, etc.).
• [ ] Integrate FAISS for vector storage and similarity search.
• [ ] On each API call, embed the user prompt and the model's response into vectors.
• [ ] Store the vectors in FAISS and store associated metadata in PostgreSQL (such as document title, conversation ID).
• [ ] Perform a similarity search using FAISS before sending a new prompt to the LLM, and include relevant history stored in PostgreSQL as context.

Phase 3: Multi-Model Routing & RAG (1-2 weeks)

• [ ] Abstract LLM connections to easily support multiple models (Gemini, DeepSeek, etc.).
• [ ] Add logic to the /chat endpoint to allow clients to specify which model to use.
• [ ] Create a separate endpoint (e.g., /add-document) to upload text files.
• [ ] Implement a RAG pipeline:
  • When a prompt is received, search FAISS for relevant vector matches and retrieve metadata from PostgreSQL.
  • Pass the relevant document chunks along with the prompt to the selected LLM.

Phase 4: Refinement and Deployment (1 week)

• [ ] Develop a simple UI (optional, could use FastAPI's built-in docs).
• [ ] Write Dockerfiles for the application.
• [ ] Add configuration management for API keys and other settings.
• [ ] Implement basic logging and error handling.

4. PostgreSQL + FAISS Workflow

• Storing Vectors: When a document is added, its vector representation is stored in FAISS. Metadata such as document titles, timestamps, and user IDs are stored in PostgreSQL.

• Querying: For a user query, embed the query into a vector. Use FAISS to perform a similarity search and retrieve the nearest vectors. Query PostgreSQL for metadata (e.g., title, author) related to the relevant vectors.

• Syncing Data: Ensure that metadata in PostgreSQL is synchronized with vectors in FAISS for accurate and consistent retrieval.