I thought I had seen every way a data import can break. Then my phone went off at 2:00 AM last Tuesday, and PagerDuty told me the daily_ingest.py script had failed again. It was not a logic error or a server crash. A vendor had switched their CSV delimiter from a comma to a pipe and told nobody, so I lost a night of sleep over a thirty-second fix. That is why I started building a self-healing data pipeline for the third-party data I do not control.
The fix itself is trivial. Open the script, swap sep=',' for sep='|', run it again. What costs you is the interrupted sleep and the effort of getting your head into a codebase while half awake. And if I can diagnose the problem by glancing at a few lines of raw text, a small language model can do the same. So I wrapped the loader in a try, heal, retry loop and let it deal with the boring exceptions on its own.
How the self-healing loop works
Most pipelines are fragile because they assume the input is clean. When that assumption breaks, the script dies. A self-healing data pipeline catches the exception instead, hands the traceback and the first few lines of the file to an LLM, and asks which parameters would work. If it gets usable parameters back, it retries the read right away.
Three libraries do the work here: Pandas loads the file, Pydantic pins down the shape of the answer, and Tenacity handles the retries. I wrote more about where this kind of automation earns its keep in pragmatic AI workflow automation.
Step 1: describing the fix with Pydantic
Ask an LLM for a delimiter and it will hand you a paragraph of chat wrapped around it. Pydantic fixes that by forcing a strict JSON schema, so the model can only return fields the code already knows how to use.
from pydantic import BaseModel, Field
from typing import Optional, Literal
# Strict schema to prevent LLM hallucinations
class CsvParams(BaseModel):
sep: str = Field(description="The delimiter, e.g. ',' or '|' or ';'")
encoding: str = Field(default="utf-8", description="File encoding")
header: Optional[int | str] = Field(default="infer", description="Row for col names")
engine: Literal["python", "c"] = "python"Step 2: the healer function
The healer only runs once something has already gone wrong. Rather than shipping a 2GB file to an API and paying for every token of it, it reads the first four lines. That is usually enough to see a wrong delimiter or a bad encoding. I have a few more notes on wiring AI into existing tooling in my WordPress AI experiments.
import openai
import json
client = openai.OpenAI()
def bb_ask_the_doctor(fp, error_trace):
print(f"🔥 Crash detected: {fp}. Analyzing...")
# Grab a small snippet. No need to blow the context window.
try:
with open(fp, "r", errors="replace") as f:
head = "".join([f.readline() for _ in range(4)])
except Exception:
head = "<<FILE UNREADABLE>>"
prompt = f"Failed to read CSV. Error: {error_trace}\nSnippet:\n{head}\nReturn JSON params."
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={ "type": "json_object" } # Using OpenAI's structured outputs
)
return json.loads(completion.choices[0].message.content)Step 3: the Tenacity retry loop
This is the part that ties the two halves together. The tenacity library lets you wrap the loader in a retry decorator, and the before_sleep hook is where the healing runs between attempts. The loader itself stays readable instead of collecting nested try/except blocks.
from tenacity import retry, stop_after_attempt, retry_if_exception_type
import pandas as pd
bb_fix_state = {}
def bb_apply_fix(retry_state):
e = retry_state.outcome.exception()
fp = retry_state.args[0]
suggestion = bb_ask_the_doctor(fp, str(e))
bb_fix_state[fp] = suggestion
@retry(
stop=stop_after_attempt(3),
retry_if_exception_type(Exception),
before_sleep=bb_apply_fix
)
def bb_tough_loader(fp):
params = bb_fix_state.get(fp, {"sep": ","})
return pd.read_csv(fp, **params)The gotchas: cost and data safety
I don’t want to oversell this, because there are real risks. Cost is the first one. If a bad deploy makes 100,000 files fail at once, you will find out about it on your API bill, so put a circuit breaker in front of the healer. The second is PII. Sensitive data should never leave for an external LLM, so if you work in healthcare or finance, run a local model like Llama-3 through Ollama. And some files should fail. When the input is corrupt or empty, the last thing you want is a model inventing a way to load garbage into your database.
If a self-healing data pipeline sounds useful but you don’t have the hours for it, I can take that work on. I have been wrestling with WordPress and awkward data integrations since the 4.x days.
Curiosity as a technical strategy
You could call using an LLM to fix a CSV overkill, and you would have a point. I still prefer an odd experiment to another night reading tracebacks in bed. What the project changed for me was the habit: I stopped treating old pipelines as things to guard and started treating them as things I could make smarter. That shift was worth more to me than the script itself.
