[{"content":"Traditional RAG has served us well. Upload your documents, chunk them, embed them, and retrieve the top-k closest matches when a user asks a question. It works. But if you\u0026rsquo;ve built enough RAG pipelines, you know the frustration: traditional RAG is blind. It doesn\u0026rsquo;t understand document structure. It can\u0026rsquo;t follow a cross-reference from a contract to its exhibits. It doesn\u0026rsquo;t know which files in a folder are even worth reading.\nI wanted something better. Something that behaves less like a search engine and more like a human researcher — someone who can skim through a stack of papers, figure out which ones matter, deep-dive into the relevant ones, and follow references when they find something interesting.\nSo I built Agentic File Query: an AI-powered document search system that replaces the fixed retrieve-and-generate pipeline with an autonomous agent that reasons about what to read and when.\nSource Code The complete source code is available on GitHub: Agentic File Query\nThe Problem with Traditional RAG Let me paint a picture. You have a folder with 20 documents — contracts, exhibits, financial reports, spreadsheets. You ask: \u0026ldquo;What is the purchase price and what are the payment terms?\u0026rdquo;\nTraditional RAG will:\nEmbed your question Find the top 5 most similar chunks across all documents Feed those chunks to the LLM Hope that the answer is somewhere in those chunks But what if the contract says \u0026ldquo;See Exhibit B for payment terms\u0026rdquo;? Traditional RAG has no idea what Exhibit B is. It can\u0026rsquo;t follow that reference. It retrieves chunks based on semantic similarity alone, with zero understanding of document relationships.\nThis is the fundamental limitation: traditional RAG has no agency. It can\u0026rsquo;t decide what to read, when to stop, or when to go back and check something it missed. It\u0026rsquo;s a one-shot pipeline.\nThe Agentic Approach: Scan → Deep Dive → Backtrack Instead of a fixed retrieval pipeline, I built this as an agent — powered by Google\u0026rsquo;s Agent Development Kit (ADK) and Gemini 3 Flash Preview. The agent has 9 tools at its disposal and follows a three-phase exploration strategy modeled on how a human researcher works:\nflowchart TD Start[\"❓ User Query\"] --\u003e P1 subgraph P1[\"Phase 1: Parallel Scan\"] S1[\"scan_folder()\"] --\u003e S2[\"Preview all documents\\n(~1 page each)\"] S2 --\u003e S3[\"Categorize each document\"] S3 --\u003e R[\"RELEVANT\"] \u0026 M[\"MAYBE\"] \u0026 SK[\"SKIP\"] end subgraph P2[\"Phase 2: Deep Dive\"] D1[\"parse_file() on\\nRELEVANT docs\"] D1 --\u003e D2[\"Extract key information\"] D2 --\u003e D3{\"Cross-references\\nfound?\"} end subgraph P3[\"Phase 3: Backtrack\"] B1[\"Explain why\\nbacktracking\"] B1 --\u003e B2[\"Parse the\\nreferenced doc\"] B2 --\u003e B3[\"Resolve all\\ncross-references\"] end R --\u003e D1 M -.-\u003e|\"if needed\"| D1 D3 --\u003e|\"Yes\"| B1 D3 --\u003e|\"No\"| Answer B3 --\u003e Answer[\"✅ Answer with Citations\"] style P1 fill:#1a1a2e,color:#e0e0e0 style P2 fill:#16213e,color:#e0e0e0 style P3 fill:#0f3460,color:#e0e0e0 Phase 1: Parallel Scan When the agent encounters a folder, it starts by calling scan_folder(). This processes every document in parallel, generating a quick preview (~1 page) of each. Think of it as quickly flipping through a stack of papers to see what\u0026rsquo;s there.\nAfter scanning, the agent categorizes each document:\nRELEVANT — clearly related to the user\u0026rsquo;s question. Gets a full read. MAYBE — could be relevant. The agent keeps these in mind. SKIP — not relevant. The agent moves on. This is a big efficiency win. Instead of blindly parsing every single document (Docling needs time to process PDFs), the agent focuses its effort where it matters.\nPhase 2: Deep Dive Next, the agent calls parse_file() on documents marked as RELEVANT. This returns the complete document content as markdown.\nWhile reading, the agent is explicitly instructed to watch for cross-references — things like \u0026ldquo;See Exhibit A/B/C…\u0026rdquo;, \u0026ldquo;As stated in the Purchase Agreement…\u0026rdquo;, document numbers, exhibit labels, and filenames. This is something traditional RAG completely misses. Cross-references are everywhere in legal documents, financial reports, and technical specs.\nPhase 3: Backtrack This is where it gets interesting. If the agent finds a cross-reference to a document it previously skipped, it backtracks:\nExplains why it\u0026rsquo;s going back (\u0026ldquo;Found a reference to Schedule B — need to check it\u0026rdquo;) Parses the referenced document Continues until all relevant cross-references are resolved sequenceDiagram participant A as Agent participant FS as Filesystem Note over A: Phase 1 — Scan A-\u003e\u003eFS: scan_folder(\"./docs\") FS--\u003e\u003eA: Previews of 8 documents Note over A: Categorize: 2 RELEVANT,1 MAYBE, 5 SKIP Note over A: Phase 2 — Deep Dive A-\u003e\u003eFS: parse_file(\"contract.pdf\") FS--\u003e\u003eA: Full contract text Note over A: Found: \"See Exhibit Bfor payment terms\" A-\u003e\u003eFS: parse_file(\"financials.xlsx\") FS--\u003e\u003eA: Financial data Note over A: Good info, no cross-refs Note over A: Phase 3 — Backtrack Note over A: \"Exhibit B was in a file Iskipped. Let me go back.\" A-\u003e\u003eFS: parse_file(\"exhibit_b.pdf\") FS--\u003e\u003eA: Payment terms Note over A: All references resolved Note over A: Compose final answerwith citations This backtracking capability is what makes the system fundamentally different from traditional RAG. The agent isn\u0026rsquo;t stuck with whatever chunks the retriever happened to find — it actively navigates the document space.\nThe 9 Tools The agent comes equipped with 9 tools, split into two groups:\nFilesystem Tools (Always Available) Tool What It Does scan_folder Previews all documents in a folder in parallel preview_file Quick look at a single file (~2-3 pages) parse_file Full document content via Docling read_text Read a plain text file directly grep_search Regex search within a file find_files Glob pattern matching in a directory Vector Search Tools (Require an Index) Tool What It Does semantic_search Cosine similarity search on indexed chunks get_indexed_document Full text of an indexed document by ID list_indexed_documents Lists all indexed documents The vector search tools light up only after you\u0026rsquo;ve run the ingestion pipeline on a folder. If no index exists, they gracefully tell the agent to fall back to filesystem tools. The agent seamlessly switches between strategies — if you\u0026rsquo;ve indexed your documents, searches start with semantic retrieval; if not, the agent does the full scan → dive → backtrack flow.\nArchitecture Here\u0026rsquo;s how the pieces fit together:\ngraph TB User[\"👤 User\"] subgraph Interfaces[\"Interfaces\"] CLI[\"CLI(Typer + Rich)\"] REST[\"FastAPI(REST + WebSocket)\"] ADK_UI[\"ADK Web UI(adk web)\"] end subgraph Agent_Layer[\"Agent Layer\"] Runner[\"ADK Runner\"] Agent[\"LlmAgent(Gemini 3 Flash Preview)\"] Tools[\"9 Tool Functions\"] end subgraph Processing[\"Document Processing\"] Parser[\"Docling Parser\"] Chunker[\"Text Chunker\"] Embeddings[\"Embedding Service(gemini-embedding-001)\"] end subgraph Storage[\"Storage Layer\"] StoreAbs[\"VectorStore Protocol\"] DuckDB[\"DuckDB + vss\"] PgVec[\"pgvector(PostgreSQL)\"] end User --\u003e CLI \u0026 REST \u0026 ADK_UI CLI \u0026 REST \u0026 ADK_UI --\u003e Runner Runner --\u003e Agent Agent --\u003e Tools Tools --\u003e Parser \u0026 StoreAbs Parser --\u003e Chunker Chunker --\u003e Embeddings Embeddings --\u003e StoreAbs StoreAbs --\u003e DuckDB \u0026 PgVec The system has four layers:\nInterfaces — Three ways in: a CLI built with Typer and Rich, a FastAPI server with WebSocket streaming for real-time events, and the built-in ADK Web UI (adk web).\nAgent Layer — The ADK Runner creates sessions and manages the agent lifecycle. The LlmAgent is powered by Gemini 3 Flash Preview, with a system prompt that encodes the three-phase strategy. The 9 tool functions are defined as simple Python functions that ADK automatically exposes to the model.\nDocument Processing — Docling handles the heavy lifting of parsing PDFs, DOCX, PPTX, XLSX, HTML, and Markdown into clean markdown text. A recursive character splitter chunks the content (1000 chars, 200 overlap), and Google\u0026rsquo;s gemini-embedding-001 generates 768-dimensional vectors.\nStorage Layer — Built around a Python Protocol (basically an interface), so swapping backends is a one-line .env change. DuckDB is the zero-setup default for local dev; pgvector on PostgreSQL is the production option.\nThe Ingestion Pipeline Before the agent can use semantic search, documents need to be indexed. The pipeline follows a four-step process:\nflowchart LR A[\"📁 Folder of\\nDocuments\"] --\u003e B[\"🔍 Find Supported\\nFiles\"] B --\u003e C[\"📄 Docling\\nParser\"] C --\u003e D[\"✂️ Recursive\\nChunker\"] D --\u003e E[\"🧮 Batch\\nEmbedding\"] E --\u003e F[\"💾 Vector\\nStore\"] style A fill:#4a9eff,color:#fff style C fill:#ff6b6b,color:#fff style D fill:#feca57,color:#333 style E fill:#48dbfb,color:#333 style F fill:#ff9ff3,color:#333 Parsing — Docling converts any supported format to clean markdown. The parser maintains a thread-safe cache keyed by filepath:mtime, so re-parsing the same file is instant.\nChunking — A recursive character splitter that tries paragraph boundaries first (\\n\\n), then newlines, sentences, words, and finally hard character splits as a last resort. The 200-character overlap ensures information at chunk boundaries is captured from both sides.\nEmbedding — Google\u0026rsquo;s gemini-embedding-001 generates 768-dimensional vectors. The service handles batching automatically (100 texts per API call), and auto-detects whether to use an API key or Vertex AI credentials.\nStoring — Each file gets a stable doc_id (SHA-256 of the path), so re-indexing a folder is idempotent — it just updates existing records.\nSwappable Storage Backends I didn\u0026rsquo;t want to lock the project into one database. The storage layer is built around a Python Protocol — a contract that says \u0026ldquo;if you implement these methods, you\u0026rsquo;re a valid vector store.\u0026rdquo;\nerDiagram CORPORA ||--o{ DOCUMENTS : \"contains\" DOCUMENTS ||--o{ CHUNKS : \"split into\" CORPORA { string corpus_id PK string folder_path UK timestamp created_at } DOCUMENTS { string doc_id PK string corpus_id FK string file_path json metadata timestamp created_at } CHUNKS { string chunk_id PK string doc_id FK string corpus_id int position string text vector embedding \"768 dimensions\" json metadata } Both backends share this three-table schema. Switching is a one-line change:\nBackend Best For Setup DuckDB Local dev, prototyping Zero setup — data stored in a single .duckdb file pgvector Production, multi-user Docker, Supabase, or Cloud SQL — just change the connection string Why Google ADK? Google\u0026rsquo;s Agent Development Kit gave me a lot for free: tool calling, session management, the adk web dev UI, and tight integration with Gemini models. Using ADK means the agent works out of the box with adk run and adk web, while also being fully programmable via the Runner for the CLI and FastAPI server.\nADK\u0026rsquo;s tool system is particularly elegant. Each tool is just a Python function with type hints — ADK inspects the signature, generates the function declaration for the model, handles the JSON marshalling, and routes the responses. No boilerplate, no adapters.\nWhy Docling? I needed something that could handle messy real-world documents — scanned PDFs, DOCX files with weird formatting, PPTX slide decks, Excel spreadsheets. Docling handles all of these and outputs clean markdown. It\u0026rsquo;s not the fastest parser out there, but the quality is consistently good, and that\u0026rsquo;s what matters when the agent needs to reason about document content.\nRunning It The system offers multiple interfaces:\n# CLI — Full agent search uv run explore explore --task \u0026#34;What is the purchase price?\u0026#34; --folder ./data/docs/ # CLI — Pre-index documents for semantic search uv run explore index --folder ./data/docs/ # CLI — Direct vector search (skips the agent) uv run explore search --query \u0026#34;purchase price\u0026#34; --folder ./data/docs/ # ADK Web UI uv run adk web src/agentic_file_query --port 8000 # FastAPI server with WebSocket streaming uv run uvicorn agentic_file_query.server:app --host 127.0.0.1 --port 8000 The CLI uses Rich for color-coded output, showing each tool call step-by-step as the agent reasons through the documents. The FastAPI server\u0026rsquo;s WebSocket endpoint streams every agent event in real time — tool calls, responses, intermediate reasoning, and the final answer — so you can build a live UI on top of it.\nKey Takeaways Building this project reinforced a few convictions:\nAgentic \u0026gt; fixed pipelines for complex document tasks. When you need multi-hop reasoning, cross-reference following, or intelligent file selection, giving the LLM agency to decide what to read changes everything.\nThe backtracking phase is critical. Without it, the agent is just a fancy scanner. The ability to recognize \u0026ldquo;I skipped something important\u0026rdquo; and go back is what makes this feel like a real researcher.\nProtocol-based abstractions pay off. The VectorStore protocol meant I could build with DuckDB for local dev and switch to pgvector for production without touching any other code. This saved real time.\nADK is remarkably ergonomic. Defining tools as plain Python functions, getting a web UI for free, and having the runner handle session management — it let me focus on the interesting parts (the strategy, the tools, the pipeline) instead of building framework glue.\nThe agentic approach isn\u0026rsquo;t a replacement for traditional RAG everywhere. For simple, well-structured knowledge bases, classic RAG is perfectly fine. But for real-world document exploration — the kind where documents reference each other, structure matters, and you don\u0026rsquo;t know upfront which files are relevant — an agent that can think, explore, and backtrack is a fundamentally better paradigm.\nTechnical Stack Agent Framework: Google ADK (Agent Development Kit) LLM: Gemini 3 Flash Preview Embeddings: gemini-embedding-001 (768d) Document Parsing: Docling Vector Storage: DuckDB (local) / pgvector (production) CLI: Typer + Rich Server: FastAPI + WebSocket Language: Python 3.12+ ","permalink":"https://sabit-shaikholla.github.io/projects/agentic-file-query/","summary":"A deep dive into Agentic File Query — an AI-powered document search built with Google ADK, Gemini, and Docling. Instead of traditional RAG, the agent uses a three-phase Scan → Deep Dive → Backtrack strategy to intelligently navigate document folders and produce cited answers.","title":"Agentic File Query: An AI Agent That Reads Documents Like a Researcher"},{"content":"In this post, I will share the detailed steps I took to set up the infrastructure for my Gemini API File Search Tool. I chose Oracle Cloud\u0026rsquo;s Always Free Tier because it offers generous resources that are perfect for hosting small to medium-sized projects without incurring costs.\nFor more details about Gemini File Search Tool, you can check my other blog post Gemini File Search Tool: RAG as a Managed Service.\nArchitecture Overview Here is a high-level view of how the components interact:\ngraph LR User[User] -- HTTPS --\u003e Nginx[Nginx Reverse Proxy] subgraph Oracle Cloud Instance Nginx -- Localhost:8000 --\u003e App[Python App] end App -- Read/Write --\u003e DB[(File Search Store)] GitHub[GitHub Repo] -- SSH Deploy --\u003e App This setup ensures a robust, secure, and cost-effective environment for hosting the Gemini API File Search Tool.\n1. Oracle Cloud Infrastructure Setup Account and Instance Creation First, I created an account on Oracle Cloud to access their Always Free services. Once logged in, I proceeded to set up the networking and compute instance.\nVCN Creation: I used the VCN Wizard to create a Virtual Cloud Network (VCN) with Internet connectivity. This automatically set up the necessary subnets, gateways, and route tables. Instance Configuration: Image: Ubuntu 22.04 Shape: VM.Standard.E2.1.Micro (This shape provides 2 vCPUs and 1GB RAM, which is eligible for the Always Free tier). SSH Keys: I generated a new SSH key pair and saved the private key to connect to the instance later. Security Rules To allow traffic to reach the server, I configured the Ingress Rules in the Default Security List associated with my public subnet:\nSSH: Allow TCP port 22 (for remote access). HTTP: Allow TCP port 80 from 0.0.0.0/0 (Anywhere). HTTPS: Allow TCP port 443 from 0.0.0.0/0 (Anywhere). Once the instance was running, I connected to it via SSH:\nssh -i /path/to/private.key ubuntu@\u0026lt;YOUR_INSTANCE_PUBLIC_IP\u0026gt; [!TIP] For a more detailed walkthrough on creating your first Linux instance, refer to the official Oracle Documentation.\n2. Server Configuration After logging in, I updated the package list and installed the essential tools and dependencies required for the project.\nsudo apt update sudo apt install -y python3 python3-venv python3-pip git Application Setup I cloned the project repository and set up a Python virtual environment to isolate the dependencies.\ncd /home/ubuntu git clone https://github.com/sabit-shaikholla/gemini-api-file-search-tool.git cd gemini-api-file-search-tool # Create and activate virtual environment python3 -m venv .venv source .venv/bin/activate # Install Python dependencies pip install --upgrade pip pip install -r requirements.txt 3. Domain and DNS Configuration To make the application accessible via a friendly URL, I used DuckDNS, a free dynamic DNS service.\nI created a domain: filequerysystem.duckdns.org. I pointed it to the Public IP address of my Oracle Cloud instance. To ensure the IP address stays updated (in case it changes), I set up a cron script. DuckDNS Update Script:\nmkdir -p ~/duckdns echo \u0026#39;echo url=\u0026#34;https://www.duckdns.org/update?domains=filequerysystem\u0026amp;token=\u0026lt;YOUR_TOKEN\u0026gt;\u0026amp;ip=\u0026#34; | curl -k -o ~/duckdns/duck.log -K -\u0026#39; \u0026gt; ~/duckdns/duck.sh chmod 700 ~/duckdns/duck.sh I then added this script to my crontab to run every 5 minutes:\n*/5 * * * * ~/duckdns/duck.sh \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 4. Nginx and Firewall Setup I used Nginx as a reverse proxy to handle incoming HTTP/HTTPS requests and forward them to the Python application.\nInstall Nginx and Configure Firewall sudo apt install -y nginx # Allow HTTP and HTTPS traffic through iptables sudo iptables -I INPUT -p tcp --dport 80 -j ACCEPT sudo iptables -I INPUT -p tcp --dport 443 -j ACCEPT sudo netfilter-persistent save Nginx Configuration I created a new configuration file for the site at /etc/nginx/sites-available/filequerysystem.\nserver { server_name filequerysystem.duckdns.org; # Increase client body size for file uploads client_max_body_size 200M; location / { proxy_pass http://127.0.0.1:8080; # Forward to the app running on port 8080 proxy_http_version 1.1; # WebSocket support (important for Streamlit) proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection \u0026#34;upgrade\u0026#34;; # Standard proxy headers proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Timeouts for long-running requests proxy_read_timeout 86400; proxy_connect_timeout 86400; proxy_send_timeout 86400; } } I enabled the site by creating a symlink:\nsudo ln -s /etc/nginx/sites-available/filequerysystem /etc/nginx/sites-enabled/ sudo nginx -t # Test configuration sudo systemctl restart nginx 5. SSL Certificate with Let\u0026rsquo;s Encrypt Security is paramount, so I secured the application with an SSL certificate using Certbot.\nsudo apt install -y certbot python3-certbot-nginx sudo certbot --nginx -d filequerysystem.duckdns.org Certbot automatically modified the Nginx configuration to force HTTPS and set up auto-renewal.\n6. Application Service To ensure the application runs continuously and restarts on failure, I created a systemd service file.\nFile: /etc/systemd/system/filequerysystem.service\n[Unit] Description=Gemini API File Search Tool After=network.target [Service] User=ubuntu Group=ubuntu WorkingDirectory=/home/ubuntu/gemini-api-file-search-tool Environment=\u0026#34;PATH=/home/ubuntu/gemini-api-file-search-tool/.venv/bin\u0026#34; ExecStart=/home/ubuntu/gemini-api-file-search-tool/.venv/bin/python app.py Restart=always [Install] WantedBy=multi-user.target Then, I enabled and started the service:\nsudo systemctl daemon-reload sudo systemctl enable filequerysystem sudo systemctl start filequerysystem 7. Automated Deployment To streamline updates, I configured a GitHub Actions workflow. This workflow automatically deploys changes to the server whenever code is pushed to the master branch.\nWorkflow File: .github/workflows/deploy.yml\nYou can check the whole file here.\nname: Deploy to Oracle Cloud on: push: branches: [ master ] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Deploy to Server uses: appleboy/ssh-action@master with: host: ${{ secrets.HOST }} username: ${{ secrets.USERNAME }} key: ${{ secrets.SSH_KEY }} script: | cd /home/ubuntu/gemini-api-file-search-tool git pull origin master source .venv/bin/activate pip install -r requirements.txt sudo systemctl restart filequerysystem ","permalink":"https://sabit-shaikholla.github.io/writing/oracle-cloud-setup-gemini-tool/","summary":"Learn how to deploy a Python application on Oracle Cloud Always Free tier, including VCN setup, security rules, Nginx reverse proxy, Let\u0026rsquo;s Encrypt SSL, and automated deployment.","title":"Deploying the Gemini File Search Tool on Oracle Cloud's Free Tier"},{"content":"I have been closely following the advancements in the Retrieval Augmented Generation (RAG) space, and from my perspective, RAG will never be the same after the introduction of the File Search Tool in the Gemini API. This new tool is a fully managed RAG system built directly into the Gemini API, effectively serving as a scalable, integrated, and highly cost-effective RAG-as-a-Service solution\nFor AI engineers and developers, this is a game-changer. Previously, building a reliable RAG pipeline was an infrastructure nightmare, requiring us to choose a vector database, develop complex chunking strategies, integrate an embedding model, and glue everything together.The File Search Tool abstracts away the entire retrieval pipeline, allowing us to focus purely on building creative applications that address user challenges.\nThe Gemini API enables RAG through the File Search tool by automatically handling the underlying complexities. It simplifies the process of grounding Gemini with your proprietary data to deliver responses that are more accurate, relevant, and verifiable.\nSource Code and Service in Action The complete source code is available on GitHub: Gemini API File Search Tool - RAG-as-a-Service\nTo experience the File Search Tool processing documents (like PDF files) and generating grounded answers directly, you can access the deployed service below: Deployed Service (on Oracle Cloud): https://filequerysystem.duckdns.org/\nFile Search: The Managed Solution How It Works: Semantic Search and Managed Indexing The core power of File Search lies in its use of semantic search. Unlike traditional keyword-based searches, semantic search understands the deeper meaning and context of your query.\nWhen you import a file, the system automatically breaks it down into chunks, embeds it using a powerful embedding model (such as gemini-embedding-001), and indexes it. These numerical representations, called embeddings, capture the semantic meaning of the text and are stored in a specialized File Search database. When a user submits a prompt, that query is also converted into an embedding, and the system performs a vector search to find the most similar and relevant document chunks from your store.\nThis entire process is consolidated into a simple workflow:\nCreate a File Search store: This store is the persistent container for your processed data and embeddings Upload and Import: You upload files, either by using the uploadToFileSearchStore API or separately using the Files API and then importFile. The data is chunked, converted into File Search embeddings, and indexed Query with File Search: You pass the FileSearch tool, referencing your store name, to the generateContent method. The model performs the semantic search and uses the retrieved context to ground its response. The following diagram illustrates the indexing and querying process internally:\ngraph TD subgraph Indexing[\"Indexing process - Offline\"] direction LR Docs[Documents] -.-\u003e FileStore(File storage) FileStore -.-\u003e EmbedModel_Idx(Embedding model) EmbedModel_Idx -.-\u003e DB[(Database)] Docs -.-\u003e EmbedModel_Idx end subgraph Querying[\"Querying process - Realtime\"] direction LR User[User] --\u003e Gemini_1[Gemini] Gemini_1 --\u003e Decision{External knowledge helpful?} Decision -- No --\u003e GenAns(Generate answer) GenAns --\u003e FinalAns_1[Final answer] Decision -- Yes --\u003e GenQuery(Generate query/ies) GenQuery --\u003e Query(Query) Query --\u003e EmbedModel_Qry(Embedding model) EmbedModel_Qry --\u003e DB DB --\u003e Context(Context) Context --\u003e Gemini_2[Gemini] Gemini_2 --\u003e FinalAns_2[Final answer] Gemini_2 --\u003e|Requires more search| GenQuery end The dotted line path (Documents straight to the Embedding model) represents the uploadToFileSearchStore API, which bypasses the File Storage step during indexing\nKey Advantages: Cost, Flexibility, and Trust Unbeatable Pricing Model The pricing structure for File Search addresses one of the biggest bottlenecks in managed RAG solutions:\nStorage is absolutely free of charge. Query time embeddings are free of charge. You only pay for embeddings at initial indexing time ($0.15 per 1 million tokens, based on the embedding model used, which is typically gemini-embedding-001). Retrieved document tokens are charged as regular context tokens. This billing approach makes the File Search Tool both significantly easier and very cost-effective to build and scale with.\nFlexibility and Management The concept of a persistent File Search store is crucial. While raw files uploaded through the Files API are typically deleted after 48 hours, the indexed data in a File Search store is stored indefinitely until you choose to delete it.\nA particularly neat feature is the ability to progressively add and remove files from an existing knowledge base. This simplifies maintaining and updating your corpus over time.\nFor fine-grained control, you can apply custom metadata (key-value pairs) to files during import. This allows you to use a metadata_filter when querying the model, enabling searches only within a specific subset of documents within a large store. You can also specify a chunking_config setting to define a maximum number of tokens per chunk and maximum number of overlapping tokens if you need more control over the chunking strategy.\nBuilt-in Verification One of the most important aspects for business use is built-in citations (autocitation). The model’s responses automatically include grounding metadata that specifies exactly which parts of your uploaded documents were used to generate the answer. This greatly simplifies the verification and fact-checking process, helping to prevent hallucination.\nLimitations and Considerations While the File Search tool simplifies RAG immensely, there are a few considerations:\nFile Size Limits: The maximum size limit per document is 100 MB. Store Size Limits: The total size of a project\u0026rsquo;s File Search stores ranges based on your user tier, from 1 GB (Free Tier) up to 1 TB (Tier 3). For optimal retrieval latencies, it is recommended to keep each individual store under 20 GB. Chunk Control: Currently, there is limited ability to adjust the number of chunks retrieved during a query. Supported Models: File Search is supported by gemini-2.5-pro and gemini-2.5-flash. Supported Formats: The tool supports a wide array of file formats, including PDF, DOCX, JSON, various text types (e.g., Markdown, HTML), and common application file types. Despite some limitations, the File Search tool eliminates the need for complex setup and management work, democratizing RAG for businesses and organizations.\n","permalink":"https://sabit-shaikholla.github.io/projects/gemini-api-file-search-tool/","summary":"An in-depth look at the Gemini API File Search Tool, a managed RAG solution that simplifies retrieval pipelines, offers semantic search, and provides built-in citations, making it easier for developers to build grounded AI applications.","title":"Gemini File Search Tool: RAG as a Managed Service"},{"content":"Large Language Models (LLMs) have demonstrated impressive capabilities, but they are not always reliable. Anyone who’s used AI chatbots has likely encountered confident yet incorrect answers – what researchers often call hallucinations. These are responses that sound plausible but are factually wrong. Such behavior poses a major challenge when deploying LLM-based agents in real-world scenarios where accuracy matters. How can we trust AI assistants if they might fabricate information?\nOne promising approach to curb LLM unreliability is Retrieval-Augmented Generation (RAG). RAG grounds an LLM’s responses in external knowledge: when asked a question, the system first retrieves relevant documents (from a database or the web) and supplies them to the LLM as context. This way, the model isn’t relying solely on its internal (potentially outdated or fuzzy) knowledge, but can base its answer on up-to-date, factual sources. RAG has quickly become a go-to pattern for improving answer accuracy. In fact, Gartner now considers RAG a key design strategy for making LLM responses more precise in enterprise AI applications.\nRAG helps, but it’s not foolproof. If the retrieval step fetches irrelevant or incorrect documents, the LLM’s answer will still be wrong, just with an illusion of support. In other words, garbage in, garbage out: an LLM will confidently use whatever context it’s given – and if that context is off-target, the result can mislead. Traditional RAG pipelines typically don’t double-check the retrieved info; they simply feed it to the model and trust it blindly. This is where new strategies come in to inject a layer of verification and correction into the process.\nFrom RAG to Corrective RAG: Self-Checking Retrieval for Accuracy One such strategy is Corrective Retrieval-Augmented Generation (CRAG), often shortened to Corrective RAG. This advanced technique builds upon RAG by adding an in-the-loop evaluation step – essentially having the system grade its own retrieved documents before producing the final answer. You can think of it as the LLM agent pausing to ask: “Have I actually found the information I need to answer the question?” If the retrieval results are inadequate, the agent can take corrective action (like searching again elsewhere) instead of proceeding with flawed or incomplete context.\nIn the original CRAG proposal (Yan et al., 2024), a lightweight model is used as a retrieval evaluator to assess how relevant the fetched documents are to the query. Based on this assessment, the pipeline can branch into different actions:\n✅ Relevant (High Confidence): If at least one retrieved document is clearly relevant and likely contains the answer, proceed as normal. The agent may even perform a knowledge refinement step – extracting and focusing on the most pertinent facts from those documents – before answering. ❌ Irrelevant (Low Confidence): If none of the retrieved docs seem useful (e.g. all scores fall below a low threshold), assume the retrieval failed. In this case, discard those results and try a different approach – for example, run a new web search to find better information. 🤔 Ambiguous (Medium Confidence): If the evaluation is mixed – say some content is partially relevant but not fully sufficient – take a hybrid approach. The agent can refine the initial docs (filter out noise, extract key points) and simultaneously perform an expanded search to gather additional context. The refined and new information are then combined for answer generation. This extra reflection step makes the system much more robust. Instead of blindly trusting the first retrieval, the agent actively validates and “corrects” its knowledge sources. It won’t just run with potentially irrelevant data. If the retrieved evidence is lacking, CRAG teaches the agent to recognize that and to course-correct – for instance, by searching the web for fresh info rather than risking an inaccurate answer. This approach directly targets the common failure mode of standard RAG, where a model can get led astray by bad context.\nTo illustrate, consider the flowchart below, which outlines how a Corrective RAG agent handles a user query with a self-checking retrieval process:\nflowchart TD Q[User Query] --\u003e R[Retrieve initial documents]; R --\u003e E[Evaluate relevance of docs]; E --\u003e|Relevant docs| G[Generate answer using docs]; E --\u003e|No relevant docs| W[Use web search for new info]; E --\u003e|Ambiguous| B[Refine \u0026 augment info]; W --\u003e G; B --\u003e G; G --\u003e A[Final Answer]; In a Corrective RAG pipeline, the agent evaluates the retrieved documents before answering. If the initial documents are irrelevant, it performs a new search. If they are partially relevant, it refines them and augments with additional info. Only once sufficient relevant knowledge is secured does the agent generate the answer. This self-reflective loop dramatically reduces the chance of the agent basing its answer on faulty premises.\nCrucially, CRAG doesn’t require training a gigantic new model or altering the base LLM – it’s more of an architectural pattern or workflow. The retrieval evaluator can be a relatively small model or heuristic. In practice, this means the overhead of the evaluation step is modest, especially compared to the cost of using a large LLM for the answer generation itself. The original research found that adding this reflection step significantly boosted answer accuracy across multiple tasks, outperforming both standard RAG and even a more complex iterative approach called Self-RAG. In other words, a little bit of checking goes a long way. By filtering out irrelevant info and always grounding the answer in confidently relevant knowledge, the agent produces more correct, focused responses.\nOf course, no method is without trade-offs. Corrective RAG’s extra steps do introduce some latency and complexity into the system. The agent might call an evaluator model and possibly issue a second search query, which can slow down responses. There’s also more moving parts (retriever, evaluator, etc.) to maintain. In settings where speed is paramount or resources are limited, developers need to balance this overhead against the benefit of improved reliability. That said, when accuracy is critical, the slight delay is often a worthy price for avoiding a wrong or hallucinated answer.\nBringing in OpenEvals: LLMs That Judge Themselves So how do we implement such self-checking behavior in practice? This project combines the idea of Corrective RAG with OpenEvals, an open-source toolkit for evaluating LLM outputs. OpenEvals (recently released by the LangChain team) provides ready-made LLM-as-a-judge modules – essentially prompt templates and interfaces that let one LLM critique or score another model’s output. Think of it as a unit-testing framework for AI: much like software tests validate parts of a program, OpenEvals uses language models to validate parts of an AI agent’s behavior. OpenEvals comes with a suite of evaluation prompts for common criteria in LLM applications. For example, it includes built-in evaluators for:\nCorrectness – Does the model’s answer match a known correct answer (ground truth)? Helpfulness – Does the answer address the user’s question effectively and fully? Groundedness – Is the answer grounded in the provided context documents (or is it introducing unsupported facts)? Retrieval Relevance – Are the retrieved documents actually relevant to the query? In my reliability-focused agent, I used several of these evaluators in the loop as reflection mechanisms. The retrieval relevance check is especially key: after the agent retrieves candidate documents via a search tool, it immediately invokes an OpenEvals prompt to judge how well each document answers the user’s question. If a document is deemed irrelevant or off-topic, it can filter it out before it ever reaches the generation stage. This automatic triage of retrieval results means the LLM only sees what’s likely to help answer correctly, keeping out the “distractors” that could lead it astray.\nFor instance, suppose the user asks: “Where was the first president of FoobarLand born?” and our initial search turned up a few documents about FoobarLand (a fictional country) but none actually mention the president’s birthplace. The OpenEvals retrieval relevance evaluator would catch this mismatch. It might return a result indicating score: False, with a comment along these lines: “None of the documents specify where the first president was born\u0026hellip; the crucial information is missing. Thus, the retrieved context does not fully address the question.”. Armed with that feedback, the agent realizes its current context is insufficient to answer the question. Instead of guessing or making something up, it can dynamically trigger a broader search – for example, querying an online encyclopedia or database for the president’s bio. This is the Corrective RAG behavior in action: using the evaluator’s critique to decide the next step (a new retrieval) rather than producing an unreliable answer.\nAgent also leverage OpenEvals to evaluate the final answer before presenting it to the user. As a reflection step, the agent generates an initial answer draft then asks an evaluator: “Is this answer actually helpful and grounded in the provided info?” If the answer did not fully address the query or included unsupported claims, the evaluator’s feedback will reflect that. The agent can then take that as a cue to refine its answer – for example, performing another targeted search for missing details, or adjusting the answer to remove unverifiable statements. This kind of corrective prompting (the agent iteratively prompting itself to improve the answer) further boosts reliability. It ensures the answer is both complete and source-backed, not just superficially plausible.\nIn summary, OpenEvals allows the LLM agent to judge its own intermediate outputs just like a human reviewer might: “Did I find the right info? Did I answer the question well?” By baking these evaluators into the agent’s decision flow, we create a feedback loop where the LLM can catch and fix its mistakes. It’s a bit meta – an AI that introspects on its own work – but extremely powerful for reducing errors.\nBuilding the Agent: LangGraph, LLMs, and Tools Source code can be found here: Github - Corrective RAG OpenEvals Implementing a Corrective RAG agent with OpenEvals involved assembling a few moving pieces. At a high level, the agent is orchestrated as a graph of actions:\nLLM Backbone: The core reasoning and answer generation is done by an LLM. For flexibility and cost-efficiency, I experimented with a smaller local model (Qwen-2.5, a 7B parameter model) running via the Ollama engine, as well as a larger cloud model (Google’s Gemini 2.5) for comparison. The agent code can work with any LLM that LangChain/LangGraph supports – you just plug in the model API or runtime of choice. The local Qwen model is fast for prototyping, while Gemini (a state-of-the-art model) offers higher quality outputs. Web Retrieval Tool: To give the agent up-to-date knowledge, I integrated a web search capability. This uses the Tavily Search API as a tool the agent can call. Whenever the agent needs fresh information (e.g. to answer a query or to find supporting facts), it issues a search query via Tavily and gets back top relevant snippets from the web. These snippets are then treated as candidate context documents. LangGraph Orchestration: Rather than coding the entire decision logic from scratch, I took advantage of LangGraph, a framework for defining AI agent workflows as a graph of nodes. Each node can be an LLM call, a tool invocation, or a conditional logic step. LangGraph allowed me to declaratively specify the sequence: Search the web → Evaluate results → If not sufficient, search again → Generate answer → Evaluate answer. This made the complex flow easier to manage and tweak. Under the hood, LangGraph coordinates the calls to the LLM (for generation and for the OpenEvals “judge” prompts) and the search tool, passing the outputs along the chain. OpenEvals Evaluators: As described, the OpenEvals library is used to create the evaluator nodes in the graph. For example, one node uses the retrieval relevance prompt to score the search results, and another node uses a helpfulness/groundedness prompt to review the answer draft. These nodes output scores or boolean flags and explanations, which the agent logic uses to decide branches (e.g. if retrieval_relevance == False, go down the “retry search” branch). All these components together form the Corrective RAG agent. The agent’s behavior feels much more intentional and intelligent than a standard single-pass LLM. It actively seeks out the answer, verifies that its sources are useful, and isn’t afraid to loop back to searching if it realizes something important is missing. In practice, this means if you ask our agent a question, you’re more likely to get an answer that’s factually correct and well-supported by evidence, or a polite admission that it needs more information – rather than a hallucinated guess.\nResults: More Reliable Answers (and Fewer Hallucinations) By combining retrieval augmentation, self-evaluation, and iterative refinement, we significantly improved the reliability and correctness of the LLM agent’s responses. Some key outcomes and observations:\nFewer Incorrect Answers: The agent’s answers are grounded in checked information, so it’s far less prone to factual mistakes. In benchmark evaluations, approaches like CRAG have outperformed naive RAG baselines, demonstrating higher answer accuracy across both short-form Q\u0026amp;A and long-form tasks. In my tests, the agent would often catch that it didn’t have enough info to answer a tricky question and automatically go find the needed detail, rather than just winging it. Reduced Hallucination: The groundedness enforcement means the model sticks to the retrieved facts. If the model tries to introduce something not supported by the documents, the evaluator flags it, prompting the agent to verify or remove it. This feedback loop cuts down on hallucinated content. The final answers are more trustworthy because you can trace their origin to real sources, not the model’s imagination. Robustness to Retrieval Failure: Even if the first retrieval attempt misses the mark, the agent can recover. This resilience was a big advantage in experiments – when the initial search was poor, the corrective strategy ensured the overall performance didn’t drop as sharply as it would for a normal RAG system. In one scenario, the agent’s primary data source lacked an answer, but a web search via the CRAG logic found the needed info in a web search. The ability to fail gracefully and try alternatives is a hallmark of this approach. Focused and Concise Responses: By filtering out irrelevant context, the agent has a more focused knowledge base to work with. I noticed the answers tended to be more concise and on-point. The model isn’t distracted by unrelated text in the prompt, which also reduces the chance of it going off on tangents. Essentially, the self-checks act as a quality filter on the context fed to the LLM. Moderate Overhead, Clear Gains: Using these techniques does add some overhead – an extra API call here, a small model run there – but in our experience the impact on latency was acceptable. The retrieval evaluation uses a lightweight prompt (or smaller model) that is fast to execute. For many questions, the agent only needs one cycle of search and answer. The slight increase in response time is well justified by the boost in confidence one can have in the answer. When building AI solutions that might be used in business or critical domains, that confidence is priceless. Conclusion In conclusion, this project demonstrates that with the right strategies, we can tame the unpredictability of LLM agents. By fusing Corrective RAG’s self-reflective retrieval process with OpenEvals’ easy-to-use evaluators, we equip the AI with a compass and a map – it can check its direction and course-correct to stay on the path of truth. This approach turns the agent from a probabilistic matter into a more dependable problem-solver that knows when to say, “I need to double-check that.”\nAs LLMs continue to evolve, such reliability enhancements will be crucial. It’s exciting to see these techniques move from research papers into practical tools that developers can apply in their own applications. The ability for an AI agent to critique itself and iteratively improve is a powerful concept, and we’re just scratching the surface. With open-source frameworks like OpenEvals lowering the barrier, we can expect a new generation of AI agents that are not only smart but also correct – agents that earn our trust one accurate answer at a time.\nReferences Gartner Research - Key Skills to Build LLMs With Retrieval-Augmented Generation Zhen et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Yan et el., 2024, Corrective Retrieval Augmented Generation OpenEvals by LangChain ","permalink":"https://sabit-shaikholla.github.io/projects/corrective-rag-openevals/","summary":"Learn how combining Corrective RAG and OpenEvals enables LLM agents to verify their sources and answers, resulting in more accurate and reliable AI responses.","title":"Corrective RAG Agent: Self-Checking Answers with OpenEvals"},{"content":"Problem statement In the dynamic world of software development, teams constantly make choices that shape the architecture of their projects. However, the rationale behind these crucial decisions can often get lost over time, leading to confusion for new team members, repeated mistakes, and difficulty in evolving the system effectively.\nADRs are concise documents that log the context, decisions, and consequences of significant architectural choices, serving as a valuable record of a project\u0026rsquo;s design history.\nAt its core, an ADR is a document that captures a specific architectural decision, the context surrounding that decision, and its resulting consequences. These records follow a lifecycle and have different states. The collection of these ADRs forms a decision log, which provides valuable project context, including detailed design and implementation information. Team members can quickly grasp the project\u0026rsquo;s direction by skimming the headlines or delve into specific design choices by reading individual ADRs. Once an ADR is accepted, it becomes immutable; if a change is needed, a new ADR is created to supersede the old one.\nWhat exactly is an ADR? An ADR is a record of a specific architectural decision made by the team. Each ADR should clearly outline:\nThe Problem or Question: The issue or requirement that necessitated the decision. The Context: The current situation, including any relevant background information and constraints. This often includes exploring alternatives and their pros and cons. The Decision: The specific architectural choice that was made. The Rationale (Why): The reasoning behind choosing this particular solution, focusing on why this decision was made rather than how it was implemented. The Consequences: The positive and negative effects of implementing the decision, including potential risks and trade-offs. The Status: The current state of the decision (e.g., Proposed, Accepted, Rejected, Superseded). Ultimately, an ADR serves as a \u0026ldquo;diary of architectural decisions\u0026rdquo; for a project.\nWhy are ADRs Important? ADRs address a common problem in software development: the loss of architectural knowledge. Over time, the reasons behind certain design choices can become unclear, especially to new team members or even the original developers revisiting the code later. Without a documented history, teams risk:\nLosing the context of past decisions. Repeating mistakes that were already considered and avoided. Decreasing efficiency by spending time figuring out the rationale behind existing solutions. By documenting architectural decisions, ADRs act as a repository of collective memory. This brings several key benefits:\nFaster onboarding for new team members: They can quickly understand the project\u0026rsquo;s evolution and key architectural choices. Avoidance of repeated discussions: The reasoning behind a decision is readily available, preventing the team from rehashing the same topics. Improved transparency and alignment: Everyone on the team understands why and how decisions were made, fostering a shared understanding. Better quality decisions: The process of documenting forces a more thorough consideration of the decision and its implications. Easier ownership handover: When responsibilities for systems change, the new owners can quickly get up to speed by reviewing the relevant ADRs. Facilitating alignment across teams: If multiple teams are working on related projects, ADRs can help them align on best practices and avoid duplicated efforts. What constitutes an Architecturally Significant Decision? Not every small decision needs an ADR. The focus should be on architecturally significant decisions that significantly affect the software project or product. These can include choices related to:\nStructure: Architectural patterns like microservices. Non-functional requirements: Security, high availability, and fault tolerance. Dependencies: Coupling between different components. Interfaces: APIs and published contracts. Construction techniques: Libraries, frameworks, tools, and development processes. Essentially, if a decision is costly to change later, it\u0026rsquo;s likely architecturally significant. While implementation details like the specific UI platform or SQL database choice are important, the decision to use a certain type of UI or database architecture would be more likely to warrant an ADR. The key is the impact and the cost of reversal.\nThe Anatomy of an ADR While specific templates can vary, a typical ADR includes the following essential components:\nTitle (or Number/Date): A concise name or a unique identifier (e.g., ADR-001) along with the date of the decision. Status: The current state of the ADR (e.g., Proposed, Accepted, Rejected, Superseded). Context (Why): A description of the problem being addressed and the surrounding circumstances that necessitate the decision. This often includes the considered alternatives, along with their pros and cons. Decision (What/How): A clear statement of the chosen architectural solution. The focus here is on what the decision is, rather than the detailed how of implementation. Rationale (Why): The justification for the chosen decision, explaining why this option was selected over the alternatives. Consequences: A description of the potential positive and negative outcomes of the decision, including any trade-offs. Timestamp: Recording when additions or modifications are made to the ADR, especially relevant for aspects that might change over time. Stakeholders (Optional): Listing the individuals or teams affected by the decision. The focus on the reasoning behind the decision (\u0026ldquo;why\u0026rdquo;) is crucial. Understanding the \u0026ldquo;why\u0026rdquo; makes it easier for others to adopt the decision and prevents future reconsideration without proper context.\nThe ADR Adoption and Review Process The process of creating and adopting ADRs typically involves these steps:\nIdentification of a Need: A team member identifies an architecturally significant decision that needs to be made. ADR Creation: An ADR owner (typically the author who is responsible for maintaining and communicating the ADR) starts writing the ADR based on a project-wide template. Proposal: The ADR owner submits the ADR in a Proposed state. Review: The ADR owner initiates a review process involving the project team. This often includes a dedicated reading time followed by a discussion of comments and questions. Feedback and Rework (if needed): If the team identifies areas for improvement, the ADR remains in the Proposed state, and the owner addresses the action points. Rejection (if applicable): The team can decide to reject the ADR, in which case the owner documents the reason for rejection and sets the state to Rejected. Acceptance: If the team approves the ADR, the owner adds a timestamp, version, and stakeholders (if applicable) and updates the state to Accepted. The accepted ADR becomes immutable. Superseding: If a previously accepted decision needs to change due to new insights, a new ADR is proposed and, if accepted, supersedes the older ADR. The state of the old ADR is then changed to Superseded. Reference: Accepted ADRs serve as a reference during code and architectural reviews to ensure that changes align with agreed-upon decisions. Every team member can propose an ADR, but establishing clear ownership is important for maintenance and communication.\ngraph TD A[Identification of a Need] --\u003e B[ADR Creation] B --\u003e C[Proposal] C --\u003e D[Review] D --\u003e E{Feedback Needed?} E -- Yes --\u003e F[Feedback and Rework] F --\u003e D E -- No --\u003e G{Accepted or Rejected?} G -- Rejected --\u003e H[Document Rejection Reason] H --\u003e I[Set State to Rejected] G -- Accepted --\u003e J[Add Timestamp and Version] J --\u003e K[Set State to Accepted] K --\u003e L[ADR Becomes Immutable] L --\u003e M[Reference During Reviews] K --\u003e N{Need to Change Decision?} N -- Yes --\u003e O[Create New ADR] O --\u003e P[Supersede Old ADR] P --\u003e Q[Set Old ADR State to Superseded] N -- No --\u003e M When Should You Write an ADR? An ADR should be written whenever a decision of significant impact is made. This includes:\nProposing large changes: Decisions that significantly affect the system\u0026rsquo;s design, maintenance, or extensibility. Proposing small but important changes: Even seemingly small decisions can have long-term implications and are worth documenting. Backfilling undocumented decisions: If an existing practice or implicit standard isn\u0026rsquo;t documented, creating an ADR can clarify it for everyone, especially new hires. The teams at Spotify found ADRs beneficial for onboarding, ownership handover, and aligning best practices across different teams and locations.\nPractical Steps for Implementing ADRs To effectively introduce ADRs into your project:\nDefine Responsible Individuals: Assign technical leads or senior developers to be primarily responsible for authoring and maintaining ADRs. Create a Simple Template: Develop an easy-to-use template that covers the essential sections. Integrate into Decision-Making: Make ADR creation a part of the process for discussing and approving significant architectural changes. Ensure Accessibility and Transparency: Store ADRs in a location easily accessible to the entire team, such as the project repository or a documentation system. Maintain Up-to-Date Documents: Regularly review ADRs for relevance and update them or create new ones when architectural changes occur. Overcoming Challenges Teams might face some resistance to adopting ADRs:\nResistance to documentation: Emphasize the benefits and start with a simple process and template. Show the team the real advantages of having a documented history. Outdated documents: Implement a process for regular review and updates, assigning responsibility for this task. Information overload: Focus on documenting only the most significant decisions and use a clear structure with tags or categories to make information easy to find. By implementing ADRs thoughtfully, teams can create a valuable decision log that fosters better understanding, reduces duplicated effort, and supports the long-term success of their projects. Various companies like Google, Spotify, Microsoft, AWS, and Red Hat have adopted similar practices, highlighting their value in managing complex systems.\nReferences ADR process - AWS Prescriptive Guidance: https://aws.amazon.com/prescriptive-guidance/documentation/adr-process/ Architectural Decision Records (ADRs) | Architectural Decision Records (GitHub): https://adr.github.io/ ADR Templates (GitHub): https://adr.github.io/templates/ When Should I Write an Architecture Decision Record - Spotify Engineering: https://engineering.atspotify.com/2020/04/14/when-should-i-write-an-architecture-decision-record/ Why you should be using architecture decision records to document your project (Red Hat Blog): https://www.redhat.com/en/blog/why-you-should-be-using-architecture-decision-records-document-your-project Architecture decision records overview (Google Cloud): https://cloud.google.com/architecture/architecture-decision-records ","permalink":"https://sabit-shaikholla.github.io/writing/adr/","summary":"Ever wondered how to keep track of critical architectural decisions in your software projects? Discover how Architecture Decision Records (ADRs) can streamline communication, prevent repeated mistakes, and ensure your team stays aligned and efficient.","title":"Documenting the 'Why': An Introduction to Architecture Decision Records"},{"content":"Project Overview In my role as a Business System Analyst, I frequently encounter organizations struggling to extract valuable insights from their document repositories. Traditional search methods often fall short when handling complex, domain-specific questions about document content.\nIn this blog post, I\u0026rsquo;ll share how I built a powerful document question-answering system using Retrieval-Augmented Generation (RAG) with Google\u0026rsquo;s Gemini 2.0 Flash model. My solution combines computer vision, natural language processing, and semantic search to create a seamless experience for users seeking information from PDF documents. By leveraging the latest advancements in multimodal AI, I\u0026rsquo;ve created a system that can:\nExtract text and visual elements from PDF documents Generate comprehensive embeddings for semantic retrieval Find the most relevant content for specific queries Produce accurate, contextually relevant answers Source Code The complete source code is available on GitHub: AI LLM Tutorials Repository\nSystem Architecture My application follows a four-stage pipeline architecture:\nDocument Ingestion: Processing uploaded PDFs and preparing them for analysis Indexing Pipeline: Extracting and encoding document content Query Processing: Understanding user questions and finding relevant content Generation Engine: Creating comprehensive, accurate responses graph TB subgraph Input [\"Document Ingestion\"] A[Document Upload] --\u003e B[Document Chunking] B --\u003e C[Vision Analysis] end subgraph Index [\"Indexing Pipeline\"] C --\u003e D[Text Extraction] D --\u003e E[Vector Embedding] E --\u003e F[(Vector Store)] end subgraph Query [\"Query Processing\"] G[User Question] --\u003e H[Query Embedding] H --\u003e I[Semantic Search] F --\u003e I I --\u003e J[Context Retrieval] end subgraph Generate [\"Generation Engine\"] J --\u003e K[Context Assembly] K --\u003e L[Gemini 2.0 Flash] L --\u003e M[Response Generation] end classDef pipeline fill:#2d2d2d,stroke:#c9c9c9,stroke-width:2px,color:#ffffff classDef storage fill:#264653,stroke:#2a9d8f,stroke-width:2px,color:#ffffff classDef process fill:#1d3557,stroke:#457b9d,stroke-width:2px,color:#ffffff class Input,Index,Query,Generate pipeline class F storage class B,C,D,E,H,I,K,L process 1. Document Ingestion The document ingestion process begins when a user uploads a PDF file through the Streamlit interface. This initial stage handles the conversion of PDFs into a format suitable for AI analysis.\ndef process_pdf(self, pdf_path: str): if not os.path.exists(pdf_path): raise FileNotFoundError(f\u0026#34;PDF file does not exist: {pdf_path}\u0026#34;) Images = PDFProcessor.pdf_to_images(pdf_path, Config.DPI) page_analyses = [] st.write(\u0026#34;Analyzing PDF pages...\u0026#34;) for i, image in enumerate(tqdm(Images)): analysis = self._analyze_image(image) if analysis: page_analyses.append(analysis) Firstly each PDF page is converted into a high-resolution image using PyMuPDF (fitz). This approach preserves the visual fidelity of the document, which is crucial for extracting information from complex layouts, tables, and figures that typical text extraction might miss.\n@staticmethod def pdf_to_images(pdf_path: str, dpi: int) -\u0026gt; List[Image.Image]: pdf_document = fitz.open(pdf_path) images = [] for page_number in range(pdf_document.page_count): page = pdf_document[page_number] pix = page.get_pixmap(matrix = fitz.Matrix(dpi / 72, dpi / 72)) image = Image.open(io.BytesIO(pix.tobytes(\u0026#34;png\u0026#34;))) images.append(image) pdf_document.close() return images Setting an appropriate DPI (dots per inch) value ensures that sufficient detail for the vision model is captured to analyze it effectively.\n2. Indexing Pipeline Once the document images are ready, the indexing pipeline takes over to transform these visuals into structured, searchable data.\nVision Analysis The heart of the indexing system relies on Gemini 2.0 Flash\u0026rsquo;s multimodal capabilities to analyze each page image:\ndef analyze_page(self, image: Image.Image) -\u0026gt; str: prompt = \u0026#34;\u0026#34;\u0026#34;Analyze this document image and: 1. Extract all visible text 2. Describe any tables, their structure and content 3. Explain any graphs or figures 4. Note any important formatting or layout details Provide a clear, detailed description that captures all key information.\u0026#34;\u0026#34;\u0026#34; return self.generate_content( model=Config.MODEL_NAME, contents=[prompt, image] ).text This approach offers significant advantages over traditional OCR:\nComprehensive text extraction: Captures text regardless of formatting or layout Table understanding: Interprets tabular data with its structure preserved Figure analysis: Explains charts, graphs, and other visual elements Layout awareness: Maintains the contextual relationship between document elements Vector Embedding Generation After extracting textual representations, vector embeddings are generated for each page using Google\u0026rsquo;s text-embedding-004 model:\nembeddings = [] try: for text in tqdm(self.data_df[\u0026#39;Analysis\u0026#39;]): embed_result = self.gemini_client.embed_content( model=Config.TEXT_EMBEDDING_MODEL_ID, contents=[text], config=types.EmbedContentConfig(task_type=\u0026#34;RETRIEVAL_DOCUMENT\u0026#34;) ) if embed_result and embed_result.embeddings: embeddings.append(embed_result.embeddings[0].values) These embeddings transform text into high-dimensional vectors that capture semantic meaning, enabling the system to find content based on conceptual similarity rather than just keyword matching.\n3. Query Processing When a user asks a question, the system processes it through a similar pipeline to match it with the most relevant document sections.\nQuery Embedding First, the user\u0026rsquo;s question is converted into the same vector space as our document embeddings:\nquery_response = self.embed_content( model=Config.TEXT_EMBEDDING_MODEL_ID, contents=[query], config=types.EmbedContentConfig(task_type=\u0026#34;RETRIEVAL_QUERY\u0026#34;) ) query_embedding = np.array(query_response.embeddings[0].values) The task_type parameter is specifically set to \u0026ldquo;RETRIEVAL_QUERY\u0026rdquo; for questions, as opposed to \u0026ldquo;RETRIEVAL_DOCUMENT\u0026rdquo; for content embeddings. This distinction optimizes the embeddings for their respective roles in the retrieval process.\nSemantic Search Once the query embedding is ready, the semantic search is performed by calculating the similarity between the query and all document pages:\nsimilarities = [ np.dot(query_embedding, np.array(page_embedding)) for page_embedding in df[\u0026#39;Embeddings\u0026#39;] ] best_idx = np.argmax(similarities) This simple dot product operation identifies the most semantically relevant content for the user\u0026rsquo;s question.\n4. Generation Engine With the relevant context identified, the final stage is to generate a comprehensive answer.\nContext Assembly The following prompt is provided that includes both the user\u0026rsquo;s question and the retrieved context:\ndef make_answer_prompt(self, query: str, passage: dict) -\u0026gt; str: escaped = passage[\u0026#39;content\u0026#39;].replace(\u0026#34;\u0026#39;\u0026#34;, \u0026#34;\u0026#34;).replace(\u0026#34;\\n\u0026#34;, \u0026#34; \u0026#34;) return textwrap.dedent(f\u0026#34;\u0026#34;\u0026#34; You are a helpful assistant analyzing research papers. Use the provided passage to answer the question. Be comprehensive but explain technical concepts clearly. If the passage is irrelevant, say so. QUESTION: \u0026#39;{query}\u0026#39; PASSAGE: \u0026#39;{escaped}\u0026#39; ANSWER: \u0026#34;\u0026#34;\u0026#34;) This structured prompt ensures the model focuses on the specific question while having access to the most relevant document content.\nResponse Generation Finally, the system uses Gemini 2.0 Flash to generate the answer:\nresponse = self.gemini_client.generate_content( model = Config.MODEL_NAME, contents = [prompt] ) The output provides a comprehensive answer based on the document context, along with citation information pointing to the specific page where the information was found.\nRate Limiting and Error Handling To ensure stable operation, I\u0026rsquo;ve implemented rate limiting using the Python ratelimit library:\n@sleep_and_retry @limits(calls=1, period=1) def create_embeddings(self, data: str): time.sleep(1) return self.client.models.embed_content( model = Config.TEXT_EMBEDDING_MODEL_ID, contents = data, config = type.EmbedContentConfig(task_type = \u0026#34;RETRIEVAL_DOCUMENT\u0026#34;) ) Additionally, the source code includes robust error handling throughout the pipeline to gracefully manage issues like empty responses, API failures, or processing errors.\nThe Streamlit Interface I\u0026rsquo;ve built a clean, user-friendly interface using Streamlit that allows users to:\nUpload PDF documents Ask questions about the document content Receive answers with source citations def main(): load_dotenv() st.set_page_config(page_title=\u0026#34;RAG AI Application\u0026#34;, page_icon=\u0026#34;📚\u0026#34;, layout=\u0026#34;wide\u0026#34;) st.title(\u0026#34;Ask the RAG AI Application\u0026#34;) # Application setup and form handling with st.form(key = \u0026#34;my_form\u0026#34;): pdf_file = st.file_uploader(\u0026#34;Upload a PDF file\u0026#34;, type = [\u0026#39;pdf\u0026#39;]) questions = st.text_input(\u0026#34;Enter your question:\u0026#34;, placeholder = \u0026#34;Please provide a short summary\u0026#34;) submit_button = st.form_submit_button(label = \u0026#34;Submit\u0026#34;) Key Benefits and Applications My RAG-based document QA system offers numerous advantages over traditional search methods:\nComprehensive understanding: Captures both textual and visual elements from documents Semantic search: Finds information based on meaning, not just keywords Contextual answers: Provides responses that directly address the user\u0026rsquo;s question Source transparency: Cites the specific document sections used to generate answers I\u0026rsquo;ve found the system is particularly valuable for:\nResearch organizations: Quickly extract insights from scientific papers and reports Legal firms: Find relevant information across large collections of legal documents Healthcare: Access and interpret medical literature and patient records Financial services: Analyze reports, prospectuses, and regulatory filings Future Enhancements While my current implementation provides impressive capabilities, I\u0026rsquo;m considering several potential enhancements that could further improve the system:\nFine-tuning: Adapting the models to specific domains or document types Chunking optimization: Experimenting with different chunking strategies Multi-document support: Extending the system to search across document repositories Streaming responses: Implementing real-time answer generation User feedback integration: Learning from user interactions to improve relevance Conclusion My RAG-based document QA system represents a significant advancement in how organizations can interact with their document repositories. By combining the power of multimodal AI with semantic search and generative models, I\u0026rsquo;ve created a tool that transforms how information is extracted from PDFs. The system demonstrates the potential of modern AI to solve real-world information access challenges, making document content more accessible and actionable than ever before.\n","permalink":"https://sabit-shaikholla.github.io/projects/ai-document-engine-rag/","summary":"A type-safe AI document analysis engine built with RAG","title":"AI Document Analysis Engine: Type-Safe RAG with Pydantic and Gemini"},{"content":"Project Overview The AI News Research Agent is an intelligent system that automates news research and synthesis using modern AI technologies. It combines Google\u0026rsquo;s Gemini 2.0 Flash model with Tavily\u0026rsquo;s search capabilities in a type-safe framework powered by Pydantic-AI.\nSystem Architecture flowchart TB subgraph Development[\"Development Environment\"] UV[UV Package Manager] --\u003e |Manages Dependencies| App end subgraph App[\"Application Core\"] UI[Streamlit UI] --\u003e Agent subgraph PydanticAI[\"Pydantic AI Framework\"] Agent[Type-Safe Agent System] --\u003e |Structured Prompts| ModelInterface ModelInterface[Model Interface] --\u003e |Type Validation| Gemini[Gemini 2.0 Flash] Agent --\u003e |Dependency Injection| Tools Tools[Tool System] --\u003e Search[Tavily Search] Results[Research Results] --\u003e |Schema Validation| Agent end Search --\u003e Agent Gemini --\u003e ModelInterface Results --\u003e UI end classDef primary fill:#4c75a6,stroke:#fff,stroke-width:2px,color:#fff classDef secondary fill:#82b1ff,stroke:#fff,stroke-width:2px,color:#fff classDef rust fill:#F74C00,stroke:#fff,stroke-width:2px,color:#fff classDef pydantic fill:#E92063,stroke:#fff,stroke-width:2px,color:#fff class UI,Results primary class ModelInterface,Tools secondary class UV rust class Agent,PydanticAI pydantic Key Features Type-Safe Architecture\nBuilt with Pydantic-AI for robust type checking Structured data validation throughout the pipeline Clear interface definitions for all components Advanced LLM Integration\nUses Google\u0026rsquo;s Gemini 2.0 Flash model Efficient prompt management Structured output generation Real-Time Search\nIntegration with Tavily Search API Configurable search parameters Asynchronous operation User-Friendly Interface\nBuilt with Streamlit Interactive parameter adjustment Clear result presentation Technical Implementation Core Components Type Definitions class ResearchResult(BaseModel): research_title: str = Field(description=\u0026#39;Markdown heading describing the article topic\u0026#39;) research_main: str = Field(description=\u0026#39;A detailed news article\u0026#39;) research_bullets: str = Field(description=\u0026#39;Key points summary\u0026#39;) @dataclass class SearchDataclass: max_results: int todays_date: str Agent Configuration search_agent = Agent( model, deps_type=ResearchDependencies, result_type=ResearchResult, system_prompt=( \u0026#34;You are a helpful research assistant and an expert in research. \u0026#34; \u0026#34;Given a single user query, you will call the \u0026#39;get_search\u0026#39; tool exactly once, \u0026#34; \u0026#34;then combine the results.\u0026#34; ) ) Search Tool Integration @search_agent.tool async def get_search(search_data: RunContext[SearchDataclass], query: str) -\u0026gt; dict: \u0026#34;\u0026#34;\u0026#34;Perform a search using the Tavily client.\u0026#34;\u0026#34;\u0026#34; results = await tavily_client.get_search_context( query=query, max_results=search_data.deps.max_results ) return json.loads(results) Technical Implementation details Dependency Injection\nClean separation of concerns Easily testable components Flexible configuration Async Operations\nNon-blocking search operations Efficient resource utilization Improved response times Type Safety\nRuntime type checking Clear interface definitions Reduced potential for errors Structured Output\nConsistent response format Validated data structures Easy integration with frontend Use Cases News Research\nQuick synthesis of current events Multi-source information gathering Automated summarization Topic Analysis\nDeep dives into specific subjects Cross-reference multiple sources Structured insights generation Trend Monitoring\nTrack emerging topics Analyze developing stories Identify key patterns Future Development Enhanced Search\nMultiple search provider support Advanced filtering options Custom search parameters Improved Processing\nAdvanced content synthesis Better source verification Enhanced summarization UI Enhancements\nMore interactive features Custom visualization options Advanced result filtering Source Code The complete source code is available on GitHub: AI LLM Tutorials Repository\nTechnical Stack Frontend: Streamlit AI Model: Google Gemini 2.0 Flash Search: Tavily API Framework: Pydantic-AI Language: Python 3.9+ Conclusion The AI News Research Agent demonstrates the practical application of modern AI technologies in creating useful tools for information gathering and synthesis. Its type-safe architecture ensures reliability while providing powerful capabilities for automated news research.\nThe project serves as an example of how to combine different AI services into a cohesive, production-ready application while maintaining code quality and type safety.\n","permalink":"https://sabit-shaikholla.github.io/projects/ai-news-agent/","summary":"A type-safe AI news research agent built with Pydantic-AI, Gemini 2.0 Flash, and Tavily Search","title":"AI News Research Agent: Pydantic-AI, Gemini and Tavily"},{"content":"Introduction Today I\u0026rsquo;ll share my experience setting up a personal website using Hugo with the PaperMod theme. This guide covers everything from installation to deployment on GitHub Pages.\nPrerequisites Before starting, ensure you have:\nGit installed GitHub account Basic command line knowledge Text editor Installation Steps 1. Install Hugo On macOS:\nbrew install hugo On Linux:\nsudo apt install hugo Verify installation:\nhugo version 2. Create a New Hugo Site Create new Hugo site\nhugo new site my-website cd my-website Initialize a Git repository\ngit init 3. Install PaperMod Theme Add PaperMod theme as a git submodule\ngit submodule add https://github.com/adityatelange/hugo-PaperMod.git themes/PaperMod Project Structure Here\u0026rsquo;s the complete structure of a Hugo website:\nmy-portfolio/ ├── archetypes/ │ └── post.md # Template for new posts ├── assets/ │ └── css/ │ └── extended/ │ └── custom.css # Custom styles ├── content/ │ ├── index.md # Homepage content │ ├── search.md # Search page │ ├── til/ # Today I Learned posts │ │ ├── _index.md │ │ └── .md │ ├── portfolio/ # Portfolio items │ │ └── _index.md │ └── random/ # Random posts │ └── _index.md ├── static/ │ └── images/ # Image files ├── themes/ │ └── PaperMod/ # PaperMod theme ├── .github/ │ └── workflows/ │ └── hugo.yml # GitHub Actions workflow ├── .gitignore ├── .gitmodules └── config.yml # Site configuration Configuration 1. Basic Configuration Create config.yml with essential settings:\nbaseURL: \u0026#34;https://username.github.io/\u0026#34; title: \u0026#34;Your Name\u0026#34; theme: [PaperMod] ... See details in PaperMod theme wiki\n2. Content Structure Create necessary directories and index files:\nmkdir -p content/{til,portfolio,random} touch content/{index,search,til/_index.md} 3. Create First Post Create a new post:\nhugo new til/first-til.md Local Development Start the development server: hugo server -D Open your browser and navigate to http://localhost:1313 to view your site Deployment to GitHub Pages Create GitHub repository: username.github.io\nConfigure GitHub Actions:\nCreate .github/workflows/hugo.yml Add deployment workflow configuration Push to GitHub:\ngit add . git commit -m \u0026#34;Initial commit\u0026#34; git remote add origin https://github.com/username/username.github.io.git git push -u origin main Key Features Implemented Dark Theme\nDefault dark theme with toggle option Custom CSS for better readability Search Functionality\nFull-text search across posts Tag-based filtering Content Organization\nTIL (Today I Learned) section Portfolio showcase Random thoughts/blog posts Navigation\nTable of Contents for posts Breadcrumbs Post navigation Customization Tips Theme Customization\nAdd custom CSS in assets/css/extended/custom.css Modify existing theme parameters in config.yml Content Management\nUse front matter for metadata Organize content with tags and categories Add images to static/images/ Conclusion Hugo with PaperMod theme provides an excellent foundation for a personal website. The combination of speed, flexibility, and ease of use makes it a great choice for developers looking to create and maintain a professional online presence.\nResources Hugo Documentation PaperMod Wiki GitHub Pages Documentation Markdown Guide ","permalink":"https://sabit-shaikholla.github.io/writing/hugo-site-generation/","summary":"A comprehensive guide on creating a personal website using Hugo static site generator with PaperMod theme, including installation, configuration, and deployment to GitHub Pages","title":"Creating a Personal Website with Hugo"},{"content":"1. Introduction Effective logging is a critical component of robust and maintainable Java applications. It provides essential insights into application behavior, facilitates issue diagnosis, and supports system monitoring and performance analysis. In this document I am outlining recommended best practices for implementing logging in Java applications to ensure clarity, efficiency, and security.\n2. Rationale for Logging Best Practices Implementing consistent and well-defined logging practices offers significant benefits, including:\nImproved System Monitoring: Logs provide real-time and historical data necessary for monitoring application health and performance. Efficient Problem Diagnosis: Detailed and contextual logs are crucial for quickly identifying and resolving issues in production and development environments. Enhanced Auditability and Security: Logging can track critical events, security-related activities, and user actions, contributing to audit trails and security compliance. Data-Driven Insights: Structured logs enable data analysis, facilitating performance optimization and identification of usage patterns. 3. Recommended Best Practices for Java Logging To achieve these benefits, the following best practices should be implemented in Java applications.\n3.1. Selection and Implementation of a Logging Framework Utilizing a dedicated logging framework is essential for managing application logs effectively. Direct use of \u0026lsquo;\u0026lsquo;\u0026lsquo;System.out.println\u0026rsquo;\u0026rsquo;\u0026rsquo; and \u0026lsquo;\u0026lsquo;\u0026lsquo;System.err\u0026rsquo;\u0026rsquo;\u0026rsquo; is discouraged for production environments due to limitations in control, flexibility, and performance.\nRecommendation: Adopt a robust and widely adopted logging framework such as Logback or Log4j2. These frameworks offer superior control over log levels, output destinations (appenders), formatting (layouts), and asynchronous logging capabilities, optimizing performance and manageability. Project standardization on a single framework is advised to ensure consistency across the application ecosystem.\n3.2. Strategic Application of Log Levels Log levels are fundamental for categorizing log messages based on severity and informational value. Consistent and appropriate use of log levels is vital for effective filtering and analysis.\nStandard Log Level Definitions and Usage:\nDEBUG: Intended for fine-grained informational events primarily useful during development and detailed troubleshooting. Examples include variable states, method entry/exit points, and detailed algorithm steps. DEBUG level logging should typically be minimized or disabled in production to reduce performance overhead. INFO: Captures general informational events that indicate normal application operation and significant milestones. Examples include application startup, service initialization, and completion of major processes. INFO level logs are valuable for high-level monitoring in all environments. WARN: Indicates potential issues or unexpected situations that do not currently impede application functionality but require attention. Examples include resource depletion warnings, non-critical configuration errors, and deprecated API usage. WARN level logs serve as an early warning system and should be monitored proactively, particularly in production. ERROR: Signifies errors that have resulted in the failure of a specific operation. The application may recover or continue processing, but an issue has occurred. Examples include exceptions caught and handled, failed service calls, and data validation errors. ERROR level logs require prompt investigation and resolution. FATAL: Denotes critical errors that are likely to lead to application termination or instability. Examples include unrecoverable exceptions, application startup failures, and resource exhaustion causing system-wide impact. FATAL errors represent emergencies necessitating immediate and critical intervention. Guidance: Establish clear guidelines within development teams regarding the appropriate use of each log level. Regular reviews should be conducted to ensure adherence to these guidelines and consistency across application modules.\n3.3. Constructing Meaningful and Contextual Log Messages The value of a log message is directly proportional to its clarity and contextual richness. Logs must provide sufficient information to diagnose and understand the logged event effectively.\nEssential Components of a Log Message:\nDescriptive Event Statement: Clearly articulate the event that occurred, moving beyond generic terms like \u0026ldquo;Error\u0026rdquo; to specific descriptions such as \u0026ldquo;Order Processing Failure.\u0026rdquo; Location Context: Include class name and method name to pinpoint the source of the log message within the codebase. Logging frameworks often provide mechanisms to automatically include this information. Causal Information (if available): Incorporate exception messages, error codes, or relevant input parameters that elucidate the reason for the event. Operational Context: Integrate identifiers relevant to the business operation, such as customer IDs, order IDs, transaction IDs, or user IDs, enabling traceability and correlation of events across system components. Example of Enhanced Log Message Construction:\ntry { // Order processing logic } catch (OrderProcessingException exception) { logger.error(\u0026#34;Order processing failure encountered for Customer ID: {}, Order ID: {}. Reason: {}\u0026#34;, customerId, orderId, exception.getMessage()); logger.debug(\u0026#34;Detailed stack trace for order processing exception:\u0026#34;, exception); // Stack trace at DEBUG level for in-depth analysis } Recommendation: Employ parameterized logging to construct log messages dynamically. This approach enhances performance by deferring string formatting until the log message is actually written, and it improves code readability.\n3.4. Implementing Structured Logging Structured logging, typically using JSON or similar formats, significantly enhances the utility of log data for automated processing and analysis.\nAdvantages of Structured Logging:\nMachine Readability and Parsability: Structured formats are readily parsed by log management and analysis tools, facilitating automated data ingestion and processing. Efficient Filtering and Searching: Structured data enables precise filtering and searching of logs based on specific attributes (e.g., log level, timestamp, application component, business identifiers). Data Analysis and Visualization: Structured logs support the creation of dashboards, reports, and visualizations for monitoring trends, identifying anomalies, and deriving operational insights. Implementation Guidance: Configure the selected logging framework to output logs in a structured format, such as JSON. Standard layouts are available in Logback and Log4j2 to achieve this.\nExample of Structured Log Output (JSON):\n{ \u0026#34;timestamp\u0026#34;: \u0026#34;2025-02-14T14:45:00.000Z\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;loggerName\u0026#34;: \u0026#34;com.electroboy06.OrderService\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Order processing encountered an error\u0026#34;, \u0026#34;customerId\u0026#34;: \u0026#34;CUST-12281996\u0026#34;, \u0026#34;orderId\u0026#34;: \u0026#34;ORD-67890\u0026#34;, \u0026#34;errorReason\u0026#34;: \u0026#34;Insufficient inventory\u0026#34;, \u0026#34;threadName\u0026#34;: \u0026#34;order-processing-thread-1\u0026#34; } 3.5. Performance Considerations in Logging While logging is essential, it introduces a performance overhead. It is crucial to implement logging practices that minimize performance impact, especially in high-throughput systems.\nPerformance Optimization Strategies:\nMinimize String Concatenation: Utilize parameterized logging to avoid unnecessary string manipulation. Avoid Computationally Intensive Operations in Log Statements: Refrain from performing complex computations or external calls solely for the purpose of constructing log messages. Employ Asynchronous Logging: Configure logging appenders to operate asynchronously, ensuring that logging operations do not block application threads. Logback and Log4j2 offer asynchronous appender options (e.g., AsyncAppender, AsyncLogger). Manage Log Levels in Production: Restrict logging in production environments to appropriate levels (INFO, WARN, ERROR, FATAL), minimizing verbose levels like DEBUG and TRACE unless actively troubleshooting a specific issue. Performance Testing Recommendation: Conduct performance testing under representative load conditions with logging enabled to quantify and mitigate any potential performance bottlenecks introduced by logging configurations.\n3.6. Security Considerations: Sensitive Data Handling in Logs Logging must be implemented with stringent security considerations, particularly concerning the accidental exposure of sensitive information.\nSecurity Best Practices:\nProhibit Logging of Sensitive Data: Never log confidential information such as passwords, credit card numbers, Personally Identifiable Information (PII), API keys, or security tokens. Data Masking and Redaction: If logging of information related to sensitive data is unavoidable for audit or tracking purposes, implement robust masking or redaction techniques to obscure sensitive portions (e.g., logging only the last four digits of a credit card number). User Input Sanitization: Exercise extreme caution when logging user-provided input, as it may inadvertently contain sensitive data. Implement input sanitization and validation processes to minimize this risk. Regular Log Audits: Establish a process for periodically auditing log files to identify and rectify any instances of unintentional sensitive data logging. Security Policy Recommendation: Develop and enforce a clear organizational policy regarding sensitive data logging, providing explicit guidelines and training to development teams.\n3.7. Centralized Log Management For applications deployed across multiple servers or microservices, centralized log management is highly recommended to streamline log aggregation, analysis, and monitoring.\nBenefits of Centralized Logging:\nConsolidated Troubleshooting: Provides a unified view of logs from across the entire application ecosystem, simplifying cross-component issue diagnosis. Enhanced Monitoring and Alerting: Facilitates the implementation of comprehensive monitoring and alerting based on aggregated log patterns, enabling proactive issue detection. Improved Security and Auditability: Centralized log repositories can be secured and audited more effectively than distributed log files, enhancing overall security posture and compliance. Centralized Logging Technologies: Consider implementing a centralized logging solution based on technologies such as the ELK stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog, or cloud-based logging services (e.g., AWS CloudWatch Logs, Google Cloud Logging, Azure Monitor Logs).\n3.8. Documentation of Logging Strategy Comprehensive documentation of the application\u0026rsquo;s logging strategy is essential for team onboarding, knowledge sharing, and maintaining consistency over time.\nDocumentation Components:\nLogging Framework Specification: Clearly identify the chosen logging framework (e.g., Logback, Log4j2) and its version. Log Level Definitions and Usage Guidelines: Document the organization\u0026rsquo;s defined log levels and provide clear guidelines for their appropriate application within the project context. Log Format Specification (Structured Logging Details): If structured logging is implemented, document the chosen format (e.g., JSON schema) and explain the meaning of key fields. Log Storage and Access Procedures: Describe where logs are stored, retention policies, and procedures for accessing and analyzing logs. Project-Specific Logging Conventions: Outline any project-specific logging conventions or best practices beyond the general guidelines. Documentation Management: Maintain the logging strategy documentation as an integral part of the application\u0026rsquo;s technical documentation, ensuring it is kept up-to-date and readily accessible to relevant teams. 3.9. Lombok @Log and @Log4j2 annotations Lombok is a Java library that allows you to reduce boilerplate code in your Java classes. It provides annotations to reduce the amount of code you need to write.\nLogger in each class:\npublic class LoggingDemo { private static final org.apache.logging.log4j.Logger log = org.apache.logging.log4j.LogManager.getLogger(LogExample.class); public static void main(final String[] args) { log.info(\u0026#34;Log something here\u0026#34;); } } With Lombok @Log annotation:\n@Log4j2 public class LoggingDemo { public static void main(final String[] args) { log.info(\u0026#34;Log something here\u0026#34;); } } 4. Conclusion Adhering to these best practices for Java logging is crucial for developing robust, maintainable, and secure applications. By strategically implementing a well-defined logging strategy, organizations can significantly improve their ability to monitor application health, diagnose issues efficiently, and gain valuable operational insights, ultimately contributing to enhanced system reliability and business performance. Consistent application of these guidelines, coupled with ongoing review and adaptation, will ensure that logging remains an effective and valuable tool throughout the application lifecycle.\n5. References Java Logging Best Practices Java Logging Basics What is the difference between Log4j, SLF4J, and Logback? Syslog 101 What is the difference between Log4j RollingFileAppender vs DailyRollingFileAppender? ","permalink":"https://sabit-shaikholla.github.io/writing/java-logging-best-practices/","summary":"Best practices for logging in Java applications, based on my experience","title":"Java Logging Best Practices"},{"content":"Project Overview This project demonstrates my implementation of a scalable e-commerce platform using microservices architecture. The solution addresses key challenges in modern e-commerce applications including modular development, independent deployability, and service isolation while maintaining a cohesive system.\nSource Code The complete source code is available on GitHub: E-Commerce Microservices Repository\nCore Technologies The platform is built using the following technologies:\nJava Spring Boot: For microservice implementation Spring Cloud Gateway: For API gateway functionality Spring Data JPA: For data access layer Spring Security: For authentication and authorization MySQL: For persistent data storage Docker \u0026amp; Docker Compose: For containerization and service orchestration Zipkin: For distributed tracing Eureka: For service discovery Swagger/OpenAPI: For API documentation Microservices Architecture Implementation Service Composition The e-commerce platform is composed of the following microservices:\ngraph TD A[API Gateway] --\u003e B[Product Service] A --\u003e C[Order Service] A --\u003e D[Inventory Service] A --\u003e E[Discovery Server] B -.-\u003e F[(Product DB)] C -.-\u003e G[(Order DB)] D -.-\u003e H[(Inventory DB)] classDef service fill:#4285F4,stroke:#333,stroke-width:1px,color:white; classDef database fill:#34A853,stroke:#333,stroke-width:1px,color:white; classDef infrastructure fill:#FBBC05,stroke:#333,stroke-width:1px,color:white; class A,B,C,D service; class F,G,H database; class E infrastructure; Each service has a specific business function:\nProduct Service: Manages product catalog and information Order Service: Handles order creation and processing Inventory Service: Manages product stock levels API Gateway: Routes client requests to appropriate services Discovery Server: Provides service registration and discovery API Gateway Implementation The API Gateway serves as the single entry point for all clients, implemented using Spring Cloud Gateway:\nRouting: Routes requests to the appropriate microservices based on path Load Balancing: Client-side load balancing for distributed instances Integration with Discovery Service: Dynamic service resolution spring: cloud: gateway: routes: - id: product-service uri: lb://product-service predicates: - Path=/api/product/** - id: order-service uri: lb://order-service predicates: - Path=/api/order/** - id: inventory-service uri: lb://inventory-service predicates: - Path=/api/inventory/** Service Discovery with Eureka The project uses Netflix Eureka for service discovery, allowing services to find and communicate with each other without hardcoded URLs:\nService Registration: Each microservice registers itself with Eureka Service Discovery: Services locate each other through the Eureka server Health Monitoring: Automatic detection of service health This implementation allows for dynamic scaling and replacement of service instances without configuration changes.\nInter-Service Communication The project implements two communication patterns:\nSynchronous Communication (REST)\nUsed for immediate responses and direct service-to-service calls Example: Checking inventory availability during order placement WebClient Integration\nNon-blocking HTTP requests between services Example: Order service communicating with inventory service // Example of WebClient usage in Order Service private final WebClient.Builder webClientBuilder; public OrderService(WebClient.Builder webClientBuilder) { this.webClientBuilder = webClientBuilder; } private Boolean checkInventory(String skuCode) { return webClientBuilder.build().get() .uri(\u0026#34;http://inventory-service/api/inventory/\u0026#34; + skuCode) .retrieve() .bodyToMono(Boolean.class) .block(); } Data Management Strategy Each microservice has its own dedicated database:\nProduct Service Database: Stores product information Order Service Database: Manages order data Inventory Service Database: Tracks product inventory levels This approach provides:\nDecoupling: Changes to one service\u0026rsquo;s data model don\u0026rsquo;t affect others Independent Scaling: Database resources can be allocated according to service needs Technology Independence: Each service can use the database technology best suited to its requirements Containerization with Docker Compose The entire application is containerized using Docker with Docker Compose for orchestration:\n# Excerpt from docker-compose.yml version: \u0026#39;3\u0026#39; services: ## MySQL Database mysql: container_name: mysql image: mysql:8.0 environment: - MYSQL_ROOT_PASSWORD=mysql - MYSQL_DATABASE=order-service ports: - \u0026#34;3306:3306\u0026#34; volumes: - ./mysql:/var/lib/mysql healthcheck: test: \u0026#34;/usr/bin/mysql --user=root --password=mysql --execute \\\u0026#34;SHOW DATABASES;\\\u0026#34;\u0026#34; interval: 2s timeout: 20s retries: 10 ## Zipkin zipkin: image: openzipkin/zipkin container_name: zipkin ports: - \u0026#34;9411:9411\u0026#34; ## Eureka Server discovery-server: image: ${DOCKER_USERNAME}/discovery-server:latest container_name: discovery-server pull_policy: always ports: - \u0026#34;8761:8761\u0026#34; environment: - SPRING_PROFILES_ACTIVE=docker depends_on: - zipkin ## API Gateway api-gateway: image: ${DOCKER_USERNAME}/api-gateway:latest container_name: api-gateway pull_policy: always ports: - \u0026#34;8181:8080\u0026#34; environment: - SPRING_PROFILES_ACTIVE=docker - LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_SECURITY=TRACE depends_on: - zipkin - discovery-server ## Product Service product-service: container_name: product-service image: ${DOCKER_USERNAME}/product-service:latest pull_policy: always environment: - SPRING_PROFILES_ACTIVE=docker depends_on: - mongo - discovery-server - api-gateway ## Order Service order-service: container_name: order-service image: ${DOCKER_USERNAME}/order-service:latest pull_policy: always environment: - SPRING_PROFILES_ACTIVE=docker depends_on: mysql: condition: service_healthy discovery-server: condition: service_started api-gateway: condition: service_started ## Inventory Service inventory-service: container_name: inventory-service image: ${DOCKER_USERNAME}/inventory-service:latest pull_policy: always environment: - SPRING_PROFILES_ACTIVE=docker depends_on: mysql: condition: service_healthy discovery-server: condition: service_started api-gateway: condition: service_started Benefits of using Docker Compose:\nInfrastructure as Code: Environment defined in version-controlled files Consistency: Identical environments across development and production Simplified Deployment: One command to start the entire system Dependency Management: Proper service startup order and health checks Distributed Tracing The project implements distributed tracing with Zipkin:\nRequest Tracking: Trace requests as they travel through multiple services Performance Monitoring: Identify bottlenecks and latency issues Error Analysis: Track where failures occur in the request chain This provides valuable insights for debugging and performance optimization in a distributed environment.\nImplementation Details by Service Product Service The Product Service manages the product catalog, implemented with:\nREST API for CRUD operations on products MongoDB for flexible document storage Spring Data MongoDB for data access abstraction Key features:\nProduct creation and retrieval Product information management Catalog organization Order Service The Order Service handles order processing with:\nTransaction Management: Ensuring order consistency Integration with Inventory: Checking stock availability before order placement MySQL Database: Relational storage for order data Order Event Generation: Creating events upon order placement Inventory Service The Inventory Service manages product stock levels:\nStock Verification API: Checking if products are in stock Inventory Updates: Adjusting stock levels as products are ordered MySQL Database: Tracking inventory quantities CI/CD Pipeline A key part of the project is the automated build and deployment process:\nGitHub Actions: Automated workflows for testing and building Docker Hub Integration: Publishing container images Automated Testing: Running tests before deployment Challenges and Solutions Challenge 1: Service Discovery in a Containerized Environment Solution: Implemented Eureka service discovery with Docker-specific configuration to handle network isolation.\nChallenge 2: Database Initialization and Migration Solution: Used Flyway for database schema migration, ensuring consistent database initialization across environments.\nChallenge 3: Inter-Service Communication Reliability Solution: Implemented proper error handling and circuit breakers for service-to-service communications to prevent cascading failures.\nChallenge 4: Container Orchestration Solution: Created a well-structured Docker Compose file with health checks and dependency management to ensure proper service startup order.\nLessons Learned Building this microservices-based e-commerce platform provided valuable insights:\nService Boundaries: Defining clear service boundaries based on business capabilities is crucial Configuration Management: Externalizing configuration for different environments simplifies deployment Docker Compose Benefits: Docker Compose provides a good balance of simplicity and power for small to medium microservices deployments Observability Importance: Implementing proper logging and tracing from the start saves significant debugging time Start Small: Begin with core services and expand gradually rather than attempting to build everything at once Conclusion This e-commerce microservices project demonstrates a practical implementation of microservices architecture using Spring Boot and Docker Compose. The approach provides modularity, independent deployability, and technology diversity while maintaining system cohesion through well-defined interfaces and service discovery.\nThe Docker Compose orchestration offers a lightweight yet powerful way to manage multiple services without the complexity of full Kubernetes orchestration, making it ideal for small to medium-sized projects or as a stepping stone toward more complex deployments.\nThe resulting platform provides a solid foundation for an e-commerce business with the flexibility to evolve individual components independently while maintaining overall system integrity.\n","permalink":"https://sabit-shaikholla.github.io/projects/ecommerce-microservices-project/","summary":"A practical implementation of a microservices-based e-commerce platform using Spring Boot, Docker Compose, and event-driven architecture","title":"E-Commerce Microservices with Spring Boot and Docker Compose"},{"content":"1. Introduction The analysis of football player performance has evolved significantly with the advent of advanced data analytics and artificial intelligence. Traditional scouting methods are increasingly supplemented by data-driven insights, enabling more informed decision-making. Football Oracle addresses this need by providing a comprehensive platform for generating and accessing detailed player analytics reports. This project aims to create a system that is not only functional but also scalable, maintainable, and secure, adhering to best practices in software engineering.\nSource Code The complete source code is available on GitHub: Football Oracle\n2. System Architecture Football Oracle follows a microservices architecture, separating the frontend and backend into distinct components. This approach enhances scalability and maintainability, allowing for independent deployment and updates.\nFrontend: Implemented using React, providing a responsive and user-friendly interface. Manages user interactions, data presentation, and communication with the backend via RESTful APIs. Backend: Developed using Spring Boot, a framework known for its rapid application development and robust ecosystem. Handles business logic, data persistence, and API endpoints. Utilizes Spring Data JPA and Hibernate for object-relational mapping (ORM). Implements JWT for secure authentication and authorization. Leverages Flyway for database migration management. Database: PostgreSQL is used for reliable and efficient data storage. The database schema is designed to accommodate player data, analytics reports, user reviews, and user profiles. External API Integration: Data is fetched from external sources such as FBREF.com. The Gemini Pro API is integrated to generate AI-driven analytics reports, providing insights into player strengths, weaknesses, and overall performance. Containerization: Docker and Docker Compose are used for containerization, enabling easy deployment and environment management. SonarQube is containerized as well, to ensure code quality. 3. Key Features and Implementation 3.1. User Authentication and Authorization Implemented using JWT to ensure secure authentication and authorization. Users can register, log in, and manage their profiles. Role-based access control is implemented to differentiate between admin, registered users, and guest users. 3.2. Player Search and Analytics Users can search for players by name. The system retrieves player data from the database or fetches it from external sources. AI-generated analytics reports are generated using the Gemini Pro API and stored in the database. 3.3. Ratings and Reviews Registered users can rate and review player analytics reports. User reviews are stored in the database and displayed alongside the reports. This feature fosters a community-driven approach to player analysis. 3.4. User Profiles Users can manage their personal information, review history, and ratings. Profiles provide a personalized experience and enhance user engagement. 4. Implementation Details 4.1. Backend Implementation RESTful API Design: The backend exposes a well-defined RESTful API using Spring MVC. Controllers are structured to handle specific resources (players, reports, users, reviews). We adhere to REST principles, using appropriate HTTP methods (GET, POST, PUT, DELETE) and status codes. API documentation is generated using Swagger UI, providing a clear interface for developers and testers. Data Persistence with Spring Data JPA and Hibernate: Spring Data JPA simplifies database interactions by providing repositories that extend JpaRepository. This eliminates the need for manual SQL queries for common operations. Spring Data JPA Hibernate, as the ORM provider, maps Java entities to database tables. Annotations like @Entity, @Id, @GeneratedValue, @OneToMany, @ManyToOne, and @JoinColumn are used to define entity relationships. Hibernate ORM Lazy loading is carefully configured to optimize performance and prevent unnecessary database queries. Transaction management is handled declaratively using @Transactional annotations, ensuring data integrity. JWT Authentication and Authorization: JWTs are used for stateless authentication. Upon successful login, the backend generates a JWT containing user information and roles. JWT Spring Security is configured to intercept incoming requests, validate JWTs, and authorize access based on user roles. Spring Security Custom filters are implemented to extract and validate JWTs from request headers. A UserDetailsService implementation is used to load user details from the database. Database Migration with Flyway: Flyway ensures that database schema changes are tracked and applied consistently across different environments. Flyway Migration scripts are stored in the src/main/resources/db/migration directory. Flyway automatically applies pending migrations during application startup. This approach simplifies database schema management and prevents inconsistencies. Logging with Logback: Logback is configured to provide comprehensive logging for debugging, monitoring, and auditing. Logback Log levels (debug, info, warn, error) are used to control the verbosity of log messages. Log messages include relevant context, such as timestamps, thread names, and class names. Log files are rotated and archived to prevent excessive disk usage. Error Handling: Custom exceptions are defined to represent specific error conditions. @ControllerAdvice is used to handle exceptions globally, providing consistent error responses to the frontend. Error responses include appropriate HTTP status codes and error messages. Gemini Pro API Integration: The backend makes HTTP requests to the Gemini Pro API to generate player analytics. Gemini Pro API The results are parsed and saved to the database. The API key is stored as an environment variable, ensuring security. The api responses are checked for errors, and handled gracefully. 4.2. Database Implementation Schema Design: The database schema is normalized to minimize data redundancy and ensure data integrity. Primary and foreign keys are used to enforce relationships between tables. Indexes are created on frequently queried columns to improve query performance. Data types are chosen carefully to optimize storage and performance. PostgreSQL Entity Relationships: Player and PlayerReport have a one-to-many relationship, with a player having multiple reports. User, Player and PlayerReport have a many to many relationship through the UserPlayerReview table. Player and PlayerStatistics have a one to many relationship. Data Retrieval Optimization: JPQL queries are used to perform complex database queries efficiently. Caching mechanisms (e.g., Hibernate\u0026rsquo;s second-level cache) are considered to reduce database load. Database connection pooling is configured to minimize connection overhead. Data Validation: Database constraints (e.g., NOT NULL, UNIQUE) are used to enforce data integrity. Backend validation is performed to ensure that data conforms to business rules. Database Security: Access to the database is restricted using strong passwords and firewall rules. Database backups are performed regularly to prevent data loss. Data migration considerations: Initial population of player data is performed through batch jobs, processing data from external web pages. Data consistency is checked after each migration. Database indexing: Indexes are used on foreign key columns, and on columns that are used in where clauses. Indexes are also used on columns that are used in join clauses. 4.3. Frontend Implementation React is used to create a single-page application (SPA). Axios is used for making HTTP requests to the backend API. Axios React Router is used for client-side routing. React Router Components are designed to be reusable and maintainable State management is handled using React\u0026rsquo;s built-in useState and useContext hooks. UI components are designed to be responsive and accessible. Form validation is implemented using libraries like Formik or React Hook Form. 5. Deployment and Configuration Docker and Docker Compose are used for containerization and deployment. Docker Docker Compose The application is deployed as a set of containers, including the backend, frontend, database, and SonarQube. Environment variables are used to configure the application, including database credentials, API keys, and JWT secrets. A CI/CD pipeline, such as GitHub Actions, could be implemented to automate build, test, and deployment processes. GitHub Actions Nginx or Apache is used as a reverse proxy to handle incoming requests and route them to the appropriate containers. 6. Evaluation and Results The system has been tested for functionality, performance, and security. Performance tests have demonstrated that the system can handle a significant load of concurrent users. Security audits have been conducted to ensure that user data is protected. SonarQube analysis ensures code quality is maintained. Unit and integration tests are written using JUnit and Mockito to ensure code reliability. 7. Future Enhancements Integration with more data sources to enhance the accuracy and comprehensiveness of analytics reports. Implementation of machine learning models for predictive analytics. Development of a mobile application for increased accessibility. Implementation of real-time data updates. Expansion of community features, such as forums and discussion boards. Implementation of a recommendation engine to suggest players based on user preferences. Enhancement of the UI/UX based on user feedback. Refactoring of the code base to improve maintainability and scalability. 8. Conclusion Football Oracle provides a scalable, AI-driven platform for football player analytics. By leveraging modern software engineering practices and technologies, this project offers a robust solution for football enthusiasts and professionals. The system\u0026rsquo;s modular design and containerized deployment make it easy to maintain and scale, ensuring its long-term viability. The integration of AI-generated analytics through the Gemini Pro API provides valuable insights that can enhance decision-making in the world of football.\n9. References Spring Boot Documentation: https://spring.io/projects/spring-boot React Documentation: https://reactjs.org/ PostgreSQL Documentation: https://www.postgresql.org/docs/ Gemini Pro API Documentation: Google Cloud AI JWT Documentation: https://jwt.io/ SonarQube Documentation: https://www.sonarqube.org/ Flyway Documentation: https://flywaydb.org/ Docker Documentation: https://docs.docker.com/ Spring Data JPA: https://spring.io/projects/spring-data-jpa Hibernate ORM: https://hibernate.org/orm/ Spring Security: https://spring.io/projects/spring-security Logback: https://logback.qos.ch/ Axios: https://axios-http.com/docs/intro React Router: https://reactrouter.com/en/main GitHub Actions: https://github.com/features/actions ","permalink":"https://sabit-shaikholla.github.io/projects/football-oracle/","summary":"A Java-based platform using Spring Boot, React, PostgreSQL, and Gemini Pro API to provide AI-driven football player analytics and insights","title":"Football Oracle: Player Analytics with Spring Boot, React and Gemini"}]