Find out more with SciForce free checklist
1. Manual embedding pipeline
The lack of automated tools for embedding transfer creates a bottleneck: embeddings are generated locally, versioned manually, and moved to production via Google Drive and manual file placement. This slows down updates, introduces room for human error, and prevents reproducible, CI-integrated secure AI deployments in healthcare.
2. Fragmented infrastructure
Limited ownership beyond the ML service and vector DB prevents the ML team from controlling or debugging full-system behavior. Integration issues, environment mismatches, and deployment delays often require cross-team coordination, slowing iteration and complicating root-cause analysis.
3. Unstructured input/output
This project addresses how to normalize unstructured clinical text with large language models to reliably interpret shorthand notes, symptom descriptions, and other free-text clinical input. Without standardized formatting or annotated data, conventional models can't be trained, and rule-based systems are brittle. Relying on LLMs introduces additional complexity: output may vary, integration becomes fragile, and downstream systems must be tightly controlled.
These challenges require flexible AI for clinical data management, including a resilient MLOps/DevOps foundation to manage variability, enforce schema constraints, and ensure consistent behavior across evolving environments.
4. Semantic QA bottlenecks
Ensuring clinical accuracy is a core challenge: even minor semantic mismatches can have significant implications. Because model output must align with subtle, context-specific interpretations, traditional automated tests fall short. Historically, every embedding or model update required manual validation by domain experts — creating a critical path delay and slowing iteration. To reduce reliance on manual expert review, the team introduced a comprehensive benchmark dataset with ground truth mappings, establishing a reliable evaluation set for further automation.
5. Scalability shift
The system initially relied on RAM-based vector search for speed, but as embedding volumes grew into multiple gigabytes, it exceeded available memory and caused instability. This forced a migration to disk-based storage, adding slight latency and requiring adjustments to indexing and retrieval under tight resource limits.
To enable reliable semantic matching from unstructured clinical input, we implemented a lightweight, modular ML integration pipeline using:
1) Azure-hosted LLMs for flexible parsing
The project uses pre-deployed Azure OpenAI models (e.g., GPT-4) to transform unstructured clinical input into structured healthcare NLP pipeline, eliminating the need for custom training or labeled datasets. This allows the system to handle highly variable user queries while keeping model maintenance centralized and scalable AI pipelines for hospitals through endpoint-based access.
2) Embedding generation handled offline
We used OpenAI model to locally generate vector embeddings for each medical terminology (e.g., SNOMED, Nometrics), ensuring full control over preprocessing and versioning. Collections are packaged as versioned archives (e.g., snomed_v1.0.zip) for easy transfer, enabling reproducible deployments and consistent behavior across environments without relying on real-time API calls.
3) Quadrant vector DB deployed in container
It uses a containerized semantic search system to serve vector similarity search via REST API, enabling fast, local retrieval of medical concepts. Embeddings are organized into named collections by terminology, allowing modular updates and scalable indexing. The container-based deployment ensures consistent behavior across environments and simplifies maintenance without relying on external vector services.
4) ML services via REST API
Project’s embedding search is delivered through a standalone, containerized Flask API that cleanly separates ML logic from the core application stack. This design allows backend systems to send free-text queries and receive structured concept matches via simple JSON calls, making integration lightweight and enabling independent updates or scaling of the ML component without affecting other services.
5) Version-Controlled Deployment
Embedding sets are packaged as versioned .zip archives and deployed across environments using containerized services with environment-scoped configuration. Each version is explicitly tracked and loaded into the local Quadrant vector database, ensuring consistent indexing and retrieval behavior across DEV and PROD. This setup enables clean separation of concerns, modular updates, and reliable promotion of embedding versions, aligning with key versioning in MLOps workflows such as reproducibility, observability, and version governance — taken further in our self-updating infection prediction pipeline, where models only reached production if they outperformed the current version.
Real-time semantic normalization
A single API endpoint accepts free-text clinical input, transforms it into embeddings combining LLM normalization with local vector databases in healthcare. Later, it performs embedding-based information retrieval against curated, versioned medical terminologies in the Quadrant database. The API returns ranked, standardized concepts with codes, labels, and confidence scores, enabling fast, consistent AI-powered terminology mapping of unstructured input to structured clinical concepts.
Support for multiple terminologies
The platform supports semantic search across SNOMED, LOINC, and RxNorm embeddings to retrieve standardized medical concepts with high precision. Each collection is independently maintained, versioned, and accessible via API, enabling semantic search in healthcare without retraining or reconfiguration.
Operational Version Introspection
A dedicated API endpoint exposes the currently loaded versions of terminology collections, embedding sets, and LLM configurations. This enables engineers and QA teams to quickly verify deployment state, troubleshoot inconsistencies across environments, and ensure alignment during testing and release validation—without accessing infrastructure directly.
Human-in-the-loop validation
To ensure semantic accuracy, domain experts created a benchmark dataset of 200 mapped clinical concepts with validated ground truth. This allows consistent testing of different models and prompt configurations without requiring repeated expert review. While expert oversight remains critical before release, the benchmark enables partial automation of QA and supports more efficient, comparative evaluation within the MLOps workflow.
1. Embedding Preparation
Each medical terminology (SNOMED CT, RxNorm, and LOINC) is processed locally to produce dense vector embeddings using embedding model, with each concept, along with its labels and synonyms, converted into a single vector. Each medical terminology (SNOMED CT, RxNorm, and LOINC) is processed locally from CSV or JSON files containing concept codes, preferred labels, and synonyms, producing a dense vector embedding for every concept to support semantic AI-driven healthcare search system — the upstream work of getting clinical data into that standardized format is covered in our PCORnet ETL pipeline on Snowflake.
Embeddings are generated in batches via local scripts that call the LLM API, respecting rate limits and error handling with exponential backoff. Each resulting vector is associated with its original concept metadata and stored in structured format. The full collection is archived by terminology and version (e.g., snomed.0.zip), enabling deterministic re-use and alignment with specific QA snapshots.
2. Versioning & Transfer
Each environment (DEV/PROD) runs a containerized stack (Flask API + Quadrant DB) with embedding archives pulled from a shared repository. Semantic versioning in MLOps and configuration are handled via .env files and Jenkins deployment. Graylog provides centralized logging and monitoring.
3. Environment Setup
The Flask-based API and Quadrant vector database are containerized and deployed via Docker Compose on dedicated virtual machines for DEV and PROD. These environments are configured independently, with no orchestration layer like Kubernetes. Each VM is configured as follows:
Local volume for embedding storage
Docker Compose stack
Environment-specific configuration via .env file
Deployment method
4. LLM Configuration
LLMs are provisioned as named deployment endpoints per environment using automated scripts integrated into the CI/CD process. The Flask-based ML service accesses these endpoints via HTTPS using the Azure SDK or REST calls, with API keys and deployment names managed through environment variables.
The Flask-based ML service accesses these models via HTTPS using Azure OpenAI SDK or direct REST calls. Authentication is handled through API keys stored in environment variables, and requests specify both the deployment name and model version.
To handle Azure rate limits (e.g., 60 RPM or token-based limits), the service includes exponential backoff with jitter and logs each retry event to Graylog. All prompts are constructed dynamically based on free-text input; no prompt templates are hardcoded.
The system performs no fine-tuning or embedding generation at runtime — all LLM usage is stateless and on-demand, with inference only.
5. Query Processing
When a user submits a free-text clinical phrase, the system processes it in real time through LLM-driven normalization and vector-based concept retrieval. This ensures accurate semantic mapping free-text clinical input to standardized medical codes. The end-to-end process involves:
Receiving the query The user sends a POST request to the /search endpoint of the Flask API, providing a JSON payload containing the raw input text (e.g., “severe shortness of breath and wheezing”).
LLM-based normalization The API forwards the text to a pre-configured Azure OpenAI endpoint (such as GPT-4). The model returns a normalized or clarified version of the input — for example, standardizing synonyms or removing irrelevant modifiers.
Embedding generation and similarity search The normalized text is embedded using a local embedding function based on OpenAI’s Ada-002 model. This vector is then compared against a selected embedding collection (e.g., snomed_v1.2) in the Quadrant vector database using cosine similarity.
Returning matched concepts The API responds with a ranked list of the most semantically relevant medical concepts, typically returning 5 to 10 matches. Each result includes the concept code, preferred label, and similarity score.
This setup allows to handle noisy clinical input and return consistent, terminology-aligned results — without requiring pre-structured queries or labeled training data.
6. Logging & Monitoring
Each query triggers structured logging from input receipt to result delivery. The Flask API logs request metadata, LLM prompts, responses, and any errors such as timeouts or Azure rate limits. Retry attempts are handled via exponential backoff in the service code and recorded with full context. Embedding generation and vector search operations are also logged with duration and status for traceability.
All logs are sent to Graylog, which is used for centralized monitoring, error tracking, and system observability.
7. QA & Validation
Each embedding update or logic change is validated in DEV using a combination of automated tests and curated clinical test cases. Automated checks verify embedding integrity, index loading, and similarity scoring behavior, while benchmark-based evaluations assess semantic accuracy across known inputs. Targeted expert review is applied when needed to ensure clinical alignment before promoting changes to PROD.
This architecture shows how to build a production-ready medical semantic search service across four medical embedding collections (e.g., SNOMED, Nometrics), each containing hundreds of thousands to several million terms.
Maintained <1 second search latency, even after shifting from in-memory to disk-based vector indexing due to RAM constraints.
The entire system runs in approximately 11 Docker containers, with the ML component isolated in 2 dedicated containers (Flask API and Quadrant DB).
Each release validated using 100+ curated test prompts, with clinical experts reviewing semantic accuracy and QA engineers confirming technical correctness.
Used GPT-4 via Azure endpoints, handling up to 120,000 tokens/minute. Retry logic with exponential backoff ensured resilience to rate limits.
No local model fine-tuning was required. Instead, runtime prompts enabled adaptable free-text clinical concept normalization across domains.
