For about two years now most of us have used LLMs on vibes. You prompt, it returns something that looks like a clean WooCommerce hook, you ship it. Looking correct and being correct are different things, though, and anyone who has shipped that code has found out the hard way. The model produces logic that holds up 90% of the time and then falls over on one race condition or one odd edge case.
Bigger models do not fix that. Formal verification does, because it forces generation into steps you can check one at a time. That is what Reinforcement Learning LLM reasoning is for. The same idea that taught AI to master Go is now pointed at how a model works through mathematical and software proofs.
The problem with guessing
Part 1 of this series covered building a proof checker. The mental model is simple: given mechanical rules, you can check whether a proof is sound. The harder question is how you train a model to follow those rules instead of predicting the next likely token. Fine-tune on a raw dataset and the model picks up the “accent” of a mathematician with none of the logic underneath.
The DeepSeek-R1 work showed what changes when the reward becomes do or do not. You hand the model a problem, let it produce several attempts, and push each attempt through a hard coded verifier. Valid logic scores 1. Hallucinations, syntax errors and logical leaps all score 0, with no partial credit for getting close.
Bootstrapping with synthetic data
An RL loop cannot start from nothing, so you need a baseline set. Ours mixes proofs translated by hand out of textbooks such as forallx with synthetic data generated by a much larger model, in our case Claude 3.5 Sonnet.
None of that synthetic data gets trusted on sight. Every proof Sonnet writes goes through the proof checker before it is allowed near the training set. The AI builds the ladder, and we test every rung before standing on it.
I wrote earlier about how the AI revolution is shifting development workflows, which sets up where this is going.
The RL loop: sample, verify, reward
On the implementation side, a service like Tinker hides most of the infrastructure, so you can run LoRA-style fine-tuning on open source models such as Qwen or GPT-OSS without fighting hardware. The training loop comes down to roughly this:
# Conceptual RL Reward Logic
def bbioon_verify_proof_reward(attempt, premises, conclusion):
# 1. Parse the generated proof
parsed_proof = proof_parser.parse(attempt)
if not parsed_proof:
return 0 # Format error
# 2. Check logical consistency against rules
verifier = ProofVerifier(rules_engine)
is_valid = verifier.verify(parsed_proof)
# 3. Ensure it actually proves what we asked
if is_valid and parsed_proof.conclusion == conclusion:
return 1
return 0Early on the model fails almost everything, and watching it is tedious work. The weights do shift, though. Nested subproofs and DeMorgan’s laws stop being patterns it imitates and become rules that, followed properly, produce the 1 it is chasing.
Where RL still struggles
Fourteen years of WordPress and WooCommerce work has taught me to distrust the word “simple,” and the RL runs were no different. Models that cleared basic proofs kept stumbling on nested logic. A fine-tuned 20b model handles (not A or not B) -> not (A and B) without much trouble. Add several premises and a couple of nested contradictions and the chain falls apart.
So vibe proving is not a switch you flip. It needs better prompt optimization, curriculum learning that starts with easy proofs and ramps up, and at some point a move to a more expressive language like Lean.
If reinforcement learning and LLM reasoning are eating your dev hours, hand the work to me. I have been dealing with WordPress and awkward backend logic since the 4.x days.
What formal verification buys you
Better math is not the point. Proofs you can check are what large distributed systems are missing when you try to trust them. A model that can prove its reasoning gives you output you can verify rather than hope about, and that is a different job from generating text.
