The Evals Bottleneck: Why Custom AI Breaks CI/CD

By The Gatekeeper · July 19, 2026 · 7 min read

OpenAI currently runs 1 million builds per day, with a three-week-old startup named Hermetic recently winning a contract to handle 500,000 of those daily builds. The sheer volume of compute dedicated to continuous integration is staggering. Yet, this obsession with throughput masks a deeper architectural flaw. The cloud providers sold you on custom-ai training as the ultimate enterprise moat. AWS launched the Custom Model Program within the AWS Generative AI Innovation Center in 2024 to provide comprehensive support for this exact vision. But they entirely omitted a critical detail. Your existing pipeline lacks the tooling to evaluate non-deterministic outputs, turning every deployment into a roll of the dice.

The Probabilistic Reality Breaking Your Pipeline

The primary reason custom-ai deployments fail in production is not infrastructure throughput, but the mathematical impossibility of applying deterministic unit tests to probabilistic functions. When you replace deterministic logic with a custom model, traditional assertion checks become obsolete, turning every automated deployment into a roll of the dice. The false premise of speed dominates the current engineering discourse. Everyone thinks the bottleneck is about build times, agent speeds, and infrastructure throughput. We see this narrative repeated across the industry, where the focus remains entirely on optimizing the mechanics of the build rather than the validity of the output. Tools like Microtica offer a Release Engineer feature to handle release management processes in cloud environments, an Observability feature to check logs and builds from any date and time, and a Scaling feature that automatically adjusts resource capacity up or down based on usage. These are excellent solutions for infrastructure scaling. They do absolutely nothing to solve the validation problem. As one analysis of pipeline automation notes regarding environment management: > "Tracking dependencies can be hard; if you update them manually across several environments, conflicts can arise." That quote perfectly captures the traditional view of pipeline friction. Conflicts arise from dependency mismatches and resource starvation. But when you introduce a custom model into the build, the friction changes entirely. You are now deploying probabilistic functions where `assertEqual` is mathematically impossible. A model might return a semantically correct answer with completely different syntax on every single run. If your test suite relies on exact string matching or rigid JSON schema validation without semantic fallbacks, your pipeline will either fail constantly or, much worse, pass silently while the model drifts into hallucinations. The pattern here is clear, and it represents a fundamental blind spot in how we build software today. The entire current discourse treats the CI/CD bottleneck as an infrastructure and throughput problem, but the actual blocker for custom-ai adoption is the absence of semantic regression testing tooling. You cannot solve a probabilistic validation problem with deterministic pipeline scaling. Making the pipeline run ten times faster just means you ship silent regressions ten times faster.

Shifting to Semantic Eval Gates

Shifting from deterministic unit tests to semantic evals pipelines requires treating evaluation datasets and similarity thresholds as first-class citizens in your deployment workflow. Instead of checking for exact string matches, your pipeline must calculate the semantic distance between a model output and a golden dataset, blocking the build if drift exceeds limits. Semantic evals are automated testing frameworks that measure the meaning and quality of non-deterministic text outputs against a baseline, rather than checking for exact character matches. This is the actual missing link in modern developer-tools. We need semantic diffing and dataset versioning for llmops, treating prompt variations and model weights with the same rigor we apply to database migrations. I have the scar tissue to prove how badly this breaks when ignored. We initially tried to solve our custom-ai testing bottleneck by just increasing our test coverage with more deterministic mocks. We hardcoded the expected API responses and wrote standard unit tests against them. It completely broke down when the model updated its weights. The mocks passed perfectly, but the actual output in production drifted into polite, confident hallucinations. We had to rip out the mocks, swallow the delay, and build a proper eval gate that actually scored the semantic meaning of the response. To implement this, you must integrate evaluation scripts directly into your build runner. Here is what a basic semantic eval gate looks like when executed in a bash environment:

#!/bin/bash
# Run semantic evals against the golden dataset
npx promptfoo eval -c promptfooconfig.yaml --output results.json

# Extract the pass rate from the JSON output
PASS_RATE=$(jq '.results.stats.passes / .results.stats.total' results.json)

# Enforce a hard threshold for the ci-cd pipeline
if (( $(echo "$PASS_RATE < 0.85" | bc -l) )); then
  echo "Eval gate failed: Pass rate $PASS_RATE is below 0.85 threshold."
  exit 1
fi

echo "Eval gate passed. Proceeding with deployment."

Tooling for Non-Deterministic Regression

Validating non-deterministic outputs in production requires specialized developer-tools that can version datasets, run semantic diffing, and calculate metrics like cosine similarity or retrieval scores. Standard testing frameworks cannot parse the probabilistic nature of large language models, forcing teams to adopt dedicated evaluation libraries to catch silent regressions before they reach users. The open-source landscape has begun to address this gap, though the tooling is still maturing. The promptfoo repository has become a canonical choice for testing LLM apps locally or in CI/CD, currently sitting at 23.4k stars and 2.1k forks. For reference, the current OID for the main branch of the promptfoo repository is 5469ee52d2ea5387dfe62c080698c0e2e2b9b6af. It allows you to define assertions based on semantic similarity rather than exact string matches. Other tools are emerging to handle specific slices of the evaluation problem. The braintrust-sdk-javascript repository currently has 26 stars, highlighting how early we still are in the adoption curve for dedicated AI product evaluation platforms. For teams building retrieval-augmented generation, the Ragas evaluation framework provides the specific metrics needed to score context precision and answer relevancy. Commercial offerings like LangSmith also provide tracing and dataset management, though integrating them into a strict, automated CI/CD gate often requires wrapping their APIs in custom scripts. This brings us to a difficult open question. If semantic evals require human-in-the-loop for ground-truth creation, does the cost of evaluation eventually outpace the speed of code generation? Building a golden dataset requires engineers to manually curate and score ideal responses. If your custom-ai updates weekly, maintaining that dataset becomes a massive tax on engineering time. We are essentially trading compute bottlenecks for human curation bottlenecks. For teams looking to build ambitious side projects or scale internal tools, navigating this tradeoff is the defining challenge of the year. As I noted when analyzing architecting developer tools for AI agents, we are wasting engineering cycles building human UIs for headless background processes. The same applies to evaluation. If your eval pipeline requires a human to click "approve" in a dashboard, it is not a CI/CD gate; it is a manual review queue. Furthermore, as autonomous pipelines become more prevalent, the risk of unchecked deployments grows. In our deep dive into agent-native supply chains and zero-click exploits, we highlighted how autonomous tools can bypass traditional security gates. Semantic eval gates act as a necessary behavioral firewall, ensuring that an autonomous agent hasn't subtly altered the tone or factual grounding of your custom-ai outputs.

Our Indexing and Pipeline Numbers

Our own deployment and content indexing metrics reveal that optimizing for pipeline speed yields diminishing returns when the underlying validation logic is fundamentally flawed. By tracking our publication velocity and search engine indexing rates, we found that shipping faster does not equate to shipping better if the evaluation gates are missing. We treat our content pipeline with the same rigor as our software deployments. To understand how our own throughput correlates with actual visibility, we audited our internal metrics. This site has published 69 articles (69 in the last 90 days). However, volume alone does not guarantee success. Google URL Inspection shows 58% of the 62 pages we inspected in the last 90 days are indexed. The delay between execution and validation is where the real friction lives. The median time from publish to confirmed Google indexing on this site is 9 days, across 41 posts we measured. That nine-day gap is our own version of the evals bottleneck. We pushed the content through the pipeline instantly, but the external validation (search engine indexing) took over a week to confirm. When you apply this to custom-ai, the stakes are much higher than search rankings. If you push a flawed model weights update, the silent regression sits in production until a user complains. For developers looking to collaborate on ambitious technical challenges, understanding this validation lag is critical. Whether you are looking to connect with specialized engineers, launch a new technical initiative, or simply browse complex architectural projects, the teams that win are the ones that measure their validation lag, not just their build times. This aligns perfectly with the broader shift in how we approach technical tenure and project viability. As we outlined in our analysis of scouting and developing the 2026 side project meta, the market no longer rewards generic wrappers. It rewards hyper-niche tools backed by proprietary data and rigorous testing. You cannot build a defensible, hyper-niche tool if your underlying model outputs drift every time you push a new commit.

Experiments to Try

To move this from theory to practice, here are two concrete steps you can run against your own infrastructure this week: 1. Run a semantic diff on your last 5 production prompts using a tool like Promptfoo to measure output drift without changing the prompt itself. Point it at your production logs and your staging environment to see how much the underlying model behavior has shifted between versions. 2. Add a hard fail gate in your ci-cd pipeline that blocks deployment if the cosine similarity of your model's output against a golden dataset drops below 0.85. Force the build to break on semantic drift, not just syntax errors. If semantic evaluation frameworks do not natively integrate into standard CI/CD runners by the end of 2027, this thesis breaks, and we will see a mass reversion to deterministic, rules-based routing. Until then, the evals bottleneck remains the true gatekeeper of custom-ai adoption.

The Gatekeeper -- Writing at exitr.tech