Soundmind AI Ecosystem: AI Sales Enablement Platform

Overview

An AI Sales Enablement Platform that turns customer documents into a deployed, demo-ready RAG PoC. It analyzes the documents, generates a tailored pipeline, and deploys it, in 5 minutes instead of a 2-week engineering cycle.

What the system does

• Upload a customer document. A 3-stage pipeline profiles it automatically: Gemini 2.0 Flash analyzes first, GPT-4o reviews second.
• The AI recommends, generates, and deploys the optimal RAG pipeline from a 4-dimension strategy space: 5 chunking, 5 retrieval, 4 indexing, and 4 post-processing options.
• The sales team composes a PoC demo on the spot. The customer tries it directly in the Playground.
• One workflow automates the whole cycle: analysis, deployment, evaluation, and monitoring.
• Net effect: new engine prototype deployment lead time drops from 2 weeks to 5 minutes.

Scale

• 9 services · 161 APIs · 103+ unit tests.
• Up to 99 concurrent RAG pipelines (ports 9201 to 9299).
• 7 parser classes covering 10 file extensions (PDF, DOCX, XLSX, XLS, TXT, MD, RST, JSON, HWP, HWPX).
• LLMs: cloud APIs (OpenAI, Gemini) plus local models served with vLLM.

System Architecture

This was a solo project. I ran it end to end inside the company, from planning and design to development and deployment. At my departure the platform was in beta, with end-user testing in progress.

The ecosystem has three pillars: the customer-facing service (AI Platform), the internal operations console (AI Console: Analysis · Eval · Monitoring), and the model serving infrastructure they share.

Model serving infrastructure:

• LLM: Qwen3 series, served with vLLM behind an OpenAI-compatible API.
• Embedder / Reranker: BGE-M3 and BGE-Reranker-v2-M3, served with Infinity. vLLM misidentified BGE-M3 as XLMRobertaModel and dropped its sparse and ColBERT weights, so I moved embedding serving to Infinity, a framework specialized for embedding models.

How this page is organized:

1. RAG R&D: building an Advanced RAG, evaluating it quantitatively, running an ablation study, and arriving at the core insight that every document needs a different strategy.
2. AI Console: the internal operations platform built on those insights (Analysis · Eval · Monitoring).
3. AI Platform: the Playground where customers experience a PoC immediately (RAG Agent · Chat Agent · AICC Agent).

RAG R&D: From Advanced RAG to Ablation

I built an Advanced RAG for public-sector documents, evaluated it quantitatively, and ran an ablation study. The work produced one core insight: every document needs a different strategy. That result feeds directly into the 4-dimension decision tree of the Analysis platform.

Stage 1Advanced RAG design and build

The company had no RAG pipeline. We were entering the NLP market with a B2B2G model (company to government agency), so I decided to design an Advanced RAG from the start rather than a naive one.

Model selection (LLM · Embedder · Reranker): The platform runs on a cloud server, but the GPUs were on premise, so local model serving was mandatory. I needed a large model with reasoning ability and chose the Qwen3 series, which scored high on global benchmarks. I served it with vLLM. I first tried to unify the embedder and reranker on Qwen models as well, but vLLM deployment failed on architecture compatibility. I adopted BGE-M3 (embedder) and BGE-Reranker-v2-M3 (reranker), the open-source models with the best Korean evaluation scores, and served them with Infinity. After researching candidate techniques, I selected three core components.

Query Expansion (Rewrite + Decomposition): In a B2B2G business we cannot predict the end user's domain, but we know the customer is a government agency. Public documents demand precision. So the pipeline rewrites the same question in several forms to cover all relevant context, and decomposes compound questions into sub-queries to collect information from multiple angles.

Hybrid Search (Dense + Sparse + RRF): Government documents are full of Sino-Korean terms and agency-specific vocabulary where keyword matching matters, so I expected sparse search (BM25) to help. I combined dense and sparse retrieval with RRF. I had little experience implementing the algorithm at the time, so I adopted Weaviate, which supports hybrid search natively. Later I confirmed that metadata-filtered retrieval works better for flat documents and added Qdrant. The vector database layer is split in two by document complexity.

Reranking (top-k=5): BGE-Reranker-v2-M3 had the best Korean scores among general-purpose models, and its open-source license allowed unrestricted on-premise deployment.

Data parsing: I started with Docling, but its parsing latency was too high for practical use. I replaced it with pdfplumber plus PaddleOCR for speed. Government work then brought many HWP and HWPX files, plus DOCX and XLSX. I built UnifiedFileParser, which unifies per-extension open-source parsers: 7 parser classes, 10 extensions.

• PDFParser (.pdf): a 3-level fallback chain. pymupdf4llm first for layout-aware Markdown, pdfplumber second, PaddleOCR third for scanned Korean documents. A page with fewer than 50 characters of text is treated as scanned and routed to OCR.
• DOCXParser (.docx): python-docx, paragraphs plus tables, document properties included.
• XLSXParser (.xlsx, .xls): openpyxl (read_only, data_only), iterates all sheets and structures each one.
• TXTParser (.txt, .md, .rst): built-in, encoding detection in the order UTF-8, CP949, EUC-KR, Latin-1.
• JSONParser (.json): built-in json module, converts the JSON hierarchy to human-readable text.
• HWPParser (.hwp): olefile-based OLE compound document parsing. Extracts HWPTAG_PARA_TEXT (tag=67) records from BodyText sections with zlib decompression.
• HWPXParser (.hwpx): zipfile plus BeautifulSoup OOXML parsing. Extracts text from Contents/section*.xml.

Embedding serving benchmark: vLLM vs Infinity

After the compatibility issue pushed me to Infinity, I measured both frameworks properly: 10 PDFs (5 arXiv, 5 Korean public documents), 20 runs each. The goal was not to justify the switch after the fact. It was to map which framework fits which workload, across domains and languages, as a basis for infrastructure design.

Online serving: single-request latency and concurrent throughput (mean ± std):

vLLM wins single-request latency by 12 to 34% and serves 1.4 to 1.7x more requests per second. Infinity, however, shows a far smaller standard deviation (0.3 to 0.7 ms vs 0.4 to 3.6 ms). Its response time is consistent regardless of document type, which keeps P99 close to the mean and makes SLAs easy to predict in a production RAG pipeline.

Batch processing: throughput (batch=32, K tokens/s, mean ± std):

Infinity delivers 1.4 to 1.7x higher token throughput. It forwards a fixed batch in a single pass and maximizes GPU utilization. Long text peaked at 323K tokens/s.

Operational conclusion:

The trade-off decision: vLLM wins on single-request latency. Our business model, however, had to onboard many government agencies at once. Every new customer uploads hundreds to thousands of pages, and bulk embedding for indexing was the dominant workload. Indexing throughput, not millisecond-level query latency, was the bottleneck that decided scalability. So I chose Infinity, which leads batch throughput by 1.4 to 1.7x. Its low variance was a second benefit for operational stability. The crossover pattern reproduced consistently across all 10 documents in both languages, so the trade-off holds regardless of domain or language.

Stage 2What quantitative evaluation exposed

After building the Advanced RAG, I built a RAGAS-based evaluation framework to measure it.

LLM-generated Silver dataset: I borrowed the Medallion Architecture pattern (Bronze, Silver, Gold) from data engineering. From 3,678 pages of raw public documents (Bronze), Qwen3-VL-30B generated 81 Q&A ground-truth pairs tagged with difficulty, query type, and reasoning hops (Silver). The node-level ablation study ran on this Silver set and quantified each component's contribution (Gold). Those numbers set the decision-tree thresholds in the AI Console recommendation engine.

The limits of RAGAS: RAGAS leaned on token-overlap metrics such as ROUGE-L (0.050) and BLEU (0.002), which score answers by token-level agreement with the ground truth. Generative models phrase the same meaning differently every time, so semantically correct answers scored low. Korean made it worse: it is agglutinative, and particles and endings keep changing token forms. I switched to an LLM-as-a-Judge evaluation with 5 weighted dimensions that scores meaning instead of tokens.

Node-level experiment results:

A structural limitation the evaluations exposed: Most real queries were flat, not multi-hop. But the dominant pattern needed dozens of contexts at once. A question like "describe the project criteria for all 20 tracks" needs 20 or more contexts, and a reranker with top-k=5 cannot feed that. Analyzing these documents showed a shared trait: repetitive structure, such as per-section conclusions and requirements. This insight justified Qdrant. For repetitive documents, JSON metadata filtering beats semantic search. Store per-section metadata at upload time, and the pipeline can filter exactly the 20+ contexts it needs. The dual vector database design came from this experience: Weaviate for hybrid search, Qdrant for filtered retrieval, branched by document characteristics.

E2E evaluation: Advanced RAG scored lower. Full evaluation on 81 Q&A pairs, 5-dimension weighted average:

Completeness collapsed by 12.5 pp. For public-document Q&A, where answers must be exhaustive, that is fatal. I had to admit that the assumption "add every component and quality improves" was wrong. This failure produced the core insight: each document needs a RAG structure that fits it. The goal became a platform that generates an optimized pipeline per document.

Stage 3Finding the culprit: ablation study

E2E metrics said Advanced RAG was worse. They could not say which of the 4 added components was responsible. So I designed and ran a node-level ablation study.

Key findings:

• Removing Reranking: −12.8 pp. The single largest contributor.
• Removing Query Decomposition: +6.8 pp and 32% lower latency (129 s to 88 s). It was actively hurting quality.
• Removing Query Rewrite: −0.8 pp. Marginal contribution.
• Switching Hybrid to Dense: −5.0 pp. Mid-level contribution.

Why Query Decomposition hurt: Splitting a query dilutes its core intent. Each sub-query drags in noisy context, and answer quality drops. It also mattered that 80% of the ground truth was easy or medium difficulty: most questions were answerable from a single query without complex reasoning.

Why Hybrid lost to Dense: BM25 keyword matching was weak on Korean public documents. Korean inflection changes token forms, and the Sino-Korean terms and abbreviations of public documents fit the tokenizer poorly, so sparse search injected noise. MRR fell as the sparse weight rose: 0.838 at alpha 0.7 versus 0.811 at alpha 0.3. Hybrid still helps in domains like law and regulation where exact term matching matters, so the design branches by document type.

The core insight: E2E evaluation can show that Advanced RAG is worse. Only node-level ablation can show that Query Decomposition is the culprit. Reranking's +12.8 pp contribution was masking Query Decomposition's 6.8 pp damage. Separating the two was the point of the study.

Stage 4From insight to ecosystem design

"Every document needs a different strategy" was the R&D conclusion. It maps to design rules:

• Reranking is always on, in every pipeline.
• Query Decomposition activates only when multi-hop queries are expected (multi_hop_likelihood > 0.4).
• BM25 degrades on Korean, so retrieval branches between Dense and Hybrid on semantic importance.
• Optimal chunking depends on document structure, so the chunking strategy branches on structural complexity.

A second insight: data structuring decides answer quality. The ablation work exposed a chain: the quality of document structuring shapes semantic chunking, chunking shapes retrieval, and retrieval shapes the final answer. Text-based parsers destroy the structure of the complex tables, charts, and layouts in public documents, degrading every downstream stage. That insight started a separate research project, WigtnOCR: a LoRA fine-tune of Qwen3-VL-2B that ranked 1st in Table TEDS among 4 compared models.

Business background: why a platform. Soundmind was a voice AI company entering NLP for the first time. We had no PoC to show. Demo opportunities came through networks, and asking every prospect for a 2-week PoC window and their data was unrealistic for the sales team. When a chance appeared, the team needed to open a cloud console, upload the customer's documents, and demo on the spot. That need, combined with the R&D insights, set the goal: document analysis, tailored pipeline generation, and one-click deployment in a console that non-developers can operate. Evaluation of deployed pipelines, real-time monitoring with log analysis, and admin for users and permissions were folded into the same console. That is the Soundmind AI Ecosystem.

AI Console: Internal Operations Platform

The integrated admin console born from the R&D insights and the business need, operable by non-developers. A single Next.js web console (port 3100) runs the full RAG sales cycle: document analysis and deployment (Analysis), quality evaluation (Eval), real-time monitoring (Monitoring), and infrastructure plus user management (Admin).

Analysis: document analysis to automated RAG deployment

Upload a customer document. The AI profiles it from multiple angles, recommends the optimal RAG pipeline, and deploys it as a Docker container in one click. New engine prototype lead time: 2 weeks to 5 minutes. The architecture is 2-tier: the Analysis API (port 9200) is the control plane, and deployed pipelines (ports 9201 to 9299) are the data plane.

Design rationale: why a Dual-LLM two-stage review

A single LLM profiling 26 characteristics inherits that model's training bias. Judgments like structural complexity or domain specificity differ across models, so trusting one model risks over- or under-engineered pipelines for certain document types.

• Gemini 2.0 Flash: native PDF analysis through the File API. Handles first-pass profiling and strategy recommendation. A Flash-tier model keeps bulk analysis cost-efficient.
• GPT-4o: reviews Gemini's output. It reads the raw text extracted by the OSS parser (pdfplumber) alongside Gemini's profile, validates profile accuracy and strategy fit with Structured Outputs (json_schema strict mode), and calibrates confidence. On disagreement, GPT-4o's judgment wins.

This is not independent cross-validation. It is a sequential two-stage pipeline: Gemini analyzes, GPT-4o reviews. GPT-4o holds a comparative advantage as the reviewer because it also sees the parser-extracted text that Gemini cannot access.

Analysis criteria: 26-characteristic profiling

The criteria extend the Korean RAG strategy design of urstory-rag (a Korean-optimized production RAG open-source project) into 4 profiles and 26 characteristics tuned for public-sector documents.

• Structure profile: hierarchy depth (1 to 4+), section independence (0.0 to 1.0), cross-reference density, dominant structure type (narrative, table-centric, hierarchical, mixed, QA), repetitive patterns.
• Content profile: information density (sparse, moderate, dense), density distribution, entity relationship structure, domain specificity (0.0 to 1.0).
• Retrieval profile: expected query types (factual, comparative, aggregation, causal, procedural), keyword and semantic importance, context window need, multi-hop likelihood.
• Visual profile: table complexity (none to nested), charts, image information density, layout complexity.

Stage 1: Gemini 2.0 Flash plus text extraction, in parallel

Gemini analyzes the PDF natively and returns the 26-characteristic profile plus a strategy recommendation (Structured Output, temperature 0.1). In parallel, pdfplumber extracts the raw text for GPT review. asyncio.gather runs both at once and halves analysis time. Documents over 300 pages or 50 MB get representative page sampling: the first 60%, middle 20%, and last 20%.

Stage 2: GPT-4o review

GPT-4o validates profile accuracy (structure depth, domain specificity) and strategy fit (over- or under-engineering) in json_schema strict mode. It adjusts confidence within −0.3 to +0.2 and classifies the result as confirmed, modified, or rejected. If no GPT key is configured, the system falls back to Gemini alone with a 0.8x confidence penalty for stability.

Stage 3: strategy engine and dynamic pipeline assembly

ReportGenerator merges Gemini's original with GPT's corrections, then picks the best combination in the 4-dimension strategy space: 5 chunking, 5 retrieval, 4 indexing, and 4 post-processing options, over 6,400 theoretical combinations. The branching logic combines the ablation results with domain heuristics:

• Reranking: always on in every post-processing strategy (ablation: removal costs 12.8 pp).
• Query Decomposition: only when multi_hop_likelihood > 0.4 (ablation: unconditional use costs 6.8 pp). The 0.4 threshold is a domain heuristic marking the lowest point where single-context answers start to fail. Three tiers at 0.4, 0.5, and 0.6 route mid complexity to Multi-Query and high complexity to Graph-Enhanced.
• Retrieval: Dense when semantic_importance > 0.7, Hybrid when keyword_importance > 0.6 (ablation confirmed Korean BM25 degradation; thresholds are domain rules).
• Vector database: Qdrant for metadata-filtered strategies, Weaviate otherwise, decided automatically.

DynamicRAGPipeline assembles a LangGraph StateGraph at runtime from a ComponentRegistry. Six graph topologies: Linear, Classify-Filter, Hierarchical, Multi-Index, Graph-Enhanced, Agentic.

One-click deployment

• Jinja2 templates generate the Docker Compose configuration. Deployment is one click.
• asyncio.Lock guards concurrency. Ports auto-allocate in the 9201 to 9299 range.
• Each RAG pipeline runs isolated in its own container and registers as a service.
• A pipeline factory names each deployment from its strategy combination, for example regulatory-struct_aware-filtered-rag.

Eval: RAG quality evaluation

Quantifies the answer quality of deployed pipelines and statistically verifies the effect of strategy changes.

Design rationale: why LLM-as-a-Judge

The Eval platform productizes the evaluation framework from RAG R&D Stage 2. The 5-dimension weighted rubric and the Silver-dataset ground-truth generation moved into the console, so non-developers can evaluate any deployed pipeline immediately.

Separating the judge model from the answering model is the core design principle. When one model does both, self-bias appears. I confirmed this experimentally: the self-judging setup gave perfect scores to long answers that contained hallucinations.

LLM-as-a-Judge: 5-dimension weighted evaluation

G-Eval style rubrics with chain-of-thought reasoning. Each dimension scores on a 1 to 5 Likert scale and normalizes to 0 to 1.

A/B test: VLM parsing vs default parsing

The same document (a 69-page public AX project call for proposals) went into two pipelines. The same questions went to both. GPT-4o judged the answers.

GPT-4o Judge results:

The judge's grounds for the deductions:

• Faithfulness −0.25: the answer mixed in outside knowledge absent from the context, such as "private cost share 70%" and "in-kind contributions limited to labor cost". Hallucination.
• Completeness −0.25: it missed that the quarantine, patent information, and legislation information tracks differ on research-institute eligibility, and wrongly stated that all tracks are identical.

Conclusion: VLM parsing (WigtnOCR) provides structured Markdown context, and the LLM produces a complete answer without hallucination. Quantitatively confirmed.

A/B test statistical verification

Monitoring: real-time observability

Collects and visualizes logs from the whole ecosystem in one place, detects failures early, and shows service state in real time.

Design rationale: why Grafana + Loki

LLM serving and the RAG pipelines share the GPU server (2x RTX PRO 6000). The ELK stack's resource footprint was too heavy for that box. Grafana, Loki, and Promtail collect logs non-intrusively through the Docker socket: no application code changes, and far lighter than ELK.

Infrastructure

• Grafana 11.5.2, Loki 3.4.2, Promtail 3.4.2 for centralized log collection.
• Docker-socket-based discovery: a new container needs no configuration change.
• Labels (low cardinality, for indexing: level, platform, service) are separated from structured metadata (high cardinality: logger, module, function) to keep Loki indexing efficient.

Dashboards

• 4 dedicated dashboards: System Overview, AI Platform, Analysis, Eval.
• A separate dashboard for model serving logs.
• Covers service health, GPU allocation, pipeline management, and user and permission settings.

AI Platform: Customer-Facing Playground (B2B SaaS)

The Playground where customers experience the tailored RAG pipeline built and deployed in the AI Console, as an on-site PoC. The sales team logs into the cloud server, uploads customer documents, and demos immediately. It offers three independent agent sessions and is built on a React 19 web console and a FastAPI API gateway with JWT multi-tenant authentication and SSE streaming.

Authentication and authorization: a 5-stage evolution

Authentication started from an unauthenticated MVP and grew stepwise to PoC delivery. Multi-tenant isolation spans three layers: Company (slug), Team, and Session, keeping each customer's data separate.

1. RAG Agent: document-grounded Q&A

Answers questions over the customer's uploaded documents. The platform auto-detects and connects the pipeline that Analysis generated and deployed for that customer. Different customers get different tailored pipelines, so each one gets retrieval and answers optimized for their documents on the same platform.

• Dynamic pipeline routing: looks up each company's pipeline URL in a PipelineMapping table and routes dynamically. Up to 99 concurrent pipelines (ports 9201 to 9299), httpx.AsyncClient with lazy initialization and connection pooling.
• Graceful degradation: on pipeline failure, the request falls back to a direct LLM answer tagged with fallback metadata. User requests never dead-end.
• Retrieval Insight, 3 panels: (1) query transformation, the original query and its 5 optimized variants; (2) hybrid search scores as dense and sparse bar charts; (3) reranking impact with rank-movement badges (gold, silver, bronze). All streamed live over SSE events.
• SSE streaming: an event chain of agent_start, processing, llm_stream (token level), final_response, done. StreamingThinkParser separates think tags in real time and renders the reasoning and the answer side by side.

2. Chat Agent: autonomous tool-using agent (MVP complete)

An autonomous agent not limited to documents. The LLM assesses the situation and selects the tools to call. Built on the LangGraph ReAct architecture, it executes compound tasks step by step.

• LangGraph StateGraph: a 2-node graph of agent (LLM call) and tools (execution). Conditional routing on tool_calls, MemorySaver for session state, and a 10-iteration cap against infinite loops.
• 3 built-in tools: Tavily Search (web search, up to 5 results), File Read (TXT and PDF, with security checks blocking .env, .ssh, and .aws), and Report Agent (a document-generation sub-agent with 4 modes: summary, analysis, minutes, custom).
• MCP extension: external tools attach dynamically through a Model Context Protocol client.
• Multi-provider: OpenAI, vLLM, and Gemini behind one interface, plus Guest BYOK (bring your own key).

3. AICC Agent: AI contact center (not started at departure)

Planned as an AI contact-center agent combining the RAG Agent's document grounding with the Chat Agent's autonomous tool use, specialized for customer-support scenarios. Development had not begun by my departure in March 2026.

Roadmap at Departure

These were the plans as of March 2026, when I left. Progress after that date happened inside Soundmind without my involvement.

WigtnOCR integration: VLM-based document parsing

The goal was to extend parsing beyond text extraction to VLM parsing that preserves tables, charts, and layout. WigtnOCR, my separate research project (a LoRA fine-tune of Qwen3-VL-2B that ranked 1st in Table TEDS among 4 compared models and surpassed a 15x larger 30B model), would roll the A/B-confirmed quality gain (0.888 to 1.000) out to every pipeline.

Production release (planned)

• Stabilize UX and reliability from beta-test feedback.
• Close the loop: evaluation, analysis, and redeployment as an automatic optimization cycle.
• Build the AICC Agent and complete the three-agent lineup.