I thought I had seen every way a metadata extractor could break. Then a Slack alert came in last Friday at 4:45 PM: the nightly batch job had crashed again. My SLM for CI/CD Reliability detour started right there, on my seventh attempt to convince GPT-4 that “valid JSON” does not include Markdown code fences.
I had written “MUST” in all caps in the system prompt. To a language model. As if emphasis would land on something with no feelings and, apparently, no stable definition of what a snake_case key looks like. It did not work, so I refactored the whole approach.
The problem with “mostly consistent”
For a nightly batch pipeline feeding a data warehouse, non-determinism is not something you can shrug off. At temperature=0, GPT-4 gives you mostly consistent output. In a CI/CD context, “mostly” is another way of saying “it will break while you are on vacation.”
The failures were subtle. One night the model returned dataset_source, the next night datasetSource, and every so often source_dataset. I spent longer than I want to admit staring at JSON nulls that had come back as Python “None” strings. After 23 pipeline failures in six weeks, not one of them caused by a bug in my code, I stopped defending the setup.
Implementing a local SLM for CI/CD reliability
I went in expecting the local model to fall short on quality. What I had missed is that pulling a document into a fixed schema is not a frontier model task at all. It is structured reading comprehension, and a well-tuned 7B model like Qwen2.5 handles it once you seed it for determinism.
Ollama let me pin seed: 42, and that is the part that actually buys you reliability. Locally, the same input with the same seed returns the same output every time. Latency dropped too, from 5.8s per doc to about 1.5s once the model was loaded.
# The REST API call to Ollama ensuring determinism
def call_local_slm(doc_text):
payload = {
"model": "qwen2.5:7b-instruct",
"options": {
"temperature": 0,
"seed": 42 # This is the whole point
},
"stream": False,
"messages": [{"role": "user", "content": doc_text}]
}
# ... rest of the logicGitHub Actions and the caching problem
Moving this into a CI/CD pipeline turned up two problems I did not see coming. Service containers are network-isolated from the runner, so docker exec is out and model pulls have to go through the REST API. And a cold pull is nearly 5GB, so without a cached model your SLM for CI/CD Reliability setup adds about 4 minutes to every run.
For more on automating a dev workflow, there is my guide on Agentic AI for Repositories.
The Pydantic validation fix
Even seeded, Qwen sometimes returns “Experimental” with a capital E regardless of what the instructions say. A Pydantic field_validator with mode="before" catches that before the type check fails the build.
from pydantic import BaseModel, field_validator
class DocMetadata(BaseModel):
methodology: Literal["experimental", "review", "simulation"]
@field_validator("methodology", mode="before")
@classmethod
def normalize_input(cls, v):
return v.lower().strip() if isinstance(v, str) else vIf SLM for CI/CD reliability work is eating your dev hours, I can take it off your plate. I have been wrestling with WordPress and backend automation since the 4.x days.
The silent pipeline
The first morning the pipeline ran clean, with no notifications and no manual re-runs, I checked the logs twice. It felt wrong. I had spent months braced for the alert every time I pushed code. If you are tired of playing whack-a-mole with probabilistic output, running the model locally is worth the afternoon it takes to set up.
A local model on hardware you control is a different class of tool. It will lose to a frontier model on plenty of tasks, but it does the same thing twice, and for a nightly pipeline that is the property that counts. For technical reference, see the official Ollama API documentation or the Pydantic docs.
