The standard advice for training LLMs is still “scrape more public data,” which only makes sense if you have not looked at the quality of the modern surface web lately. I have read enough legacy codebases to know how this ends. Feed a system its own garbage and it does not get smarter, it gets weirder. That is Model Collapse in one sentence.
Fourteen years of building complex systems has convinced me that data provenance is the whole game. Right now the models are eating their own tails. AI-generated content keeps filling the public internet, and the next round of models trains on the output of the last round. Researchers call that loop Model Collapse: a degenerative process where the model gradually loses the ability to represent the tails of a distribution, then degrades into nonsense.
The surface web is exhausted
Most of the AI we use today was built on the surface web, meaning Reddit, Wikipedia and news sites. That data is noisy, heavily SEO-optimized, and by now well seeded with bot output. Solving Model Collapse means looking where the crawlers cannot reach, which is the deep web.
The deep web is not the dark web. It is the boring, high-quality material behind logins: medical portals, internal enterprise databases, verified financial records. That data is clean and authenticated, and it holds the rare edge cases that synthetic data smooths away. If you want the background on how AI maps these relationships, see my guide on decoding embedding models.
What the PROPS framework proposes
The obvious problem with deep web data is privacy. Nobody gets to scrape a hospital’s patient records. So the architecture has to change. A recent ArXiv paper introduced PROPS, short for Protected Pipelines, which pairs hardware and cryptography to bridge that gap.
- Privacy-preserving oracles act like digital notaries. They confirm the data is real without ever showing the raw bits to the AI.
- Secure enclaves are a hardware-level black box, Intel SGX being the obvious example. Training runs inside, and only the learned weights come out.
Rather than the hand-over-your-data model, PROPS builds a marketplace where you authorize specific uses of your own data. That is a real change in how AI transparency and trust could work.
A verification hook, in practice
In a WordPress context you would handle this with secure API handoffs that check data authenticity before anything reaches a processing queue. Below is a rough sketch of oracle-style verification in PHP, enough to keep garbage out of a local dataset.
<?php
/**
* Conceptual Oracle Verification Hook
* Ensures data integrity before local processing to prevent model drift.
*/
function bbioon_verify_deep_web_data( $raw_payload, $remote_signature ) {
$public_key = get_option( 'bbioon_oracle_key' );
// Verify the data was notarized by a trusted enclave
$is_valid = openssl_verify( $raw_payload, base64_decode( $remote_signature ), $public_key, OPENSSL_ALGO_SHA256 );
if ( 1 !== $is_valid ) {
error_log( 'Data verification failed. Potential training poison detected.' );
return false;
}
// Only process if the oracle testifies to the data's authenticity
return json_decode( $raw_payload, true );
}
?>Why synthetic data is not the answer
I keep hearing that we can just generate more data. Synthetic data kills diversity, though. It reinforces the middle of the bell curve, so if you have a rare medical condition or a niche technical requirement, the generator treats you as noise. In production that is the main thing driving Model Collapse.
A Nature paper already showed that recursive training leads to collapse. The way out is secure pipelines that respect privacy while still reaching the ground-truth data sitting in the deep web.
If this Model Collapse stuff is eating your dev hours, hand it to me. I have been wrestling with WordPress since the 4.x days.
Where this leaves us
The data crisis is not a shortage of information. It is a shortage of trust and infrastructure. There is plenty of data left to build the next generation of AI on, but it is locked away, and PROPS with secure enclaves is a credible key. Stop training on garbage and your models will stop turning into it.
