We need to talk about LLM document summarization. For some reason, the standard advice for handling massive files has become “just chunk it and hope for the best,” but if you have ever tried to summarize a 200,000-word handbook, you know that is a recipe for hallucinations and context drift. In my 14 years of development, I have learned that brute force rarely scales. When your context window is a bottleneck, you don’t need a bigger hammer; you need a smarter filter.
If you missed Part 1, we tackled the “Lost in the Middle” problem by using K-means to group document chunks into semantic folders. Now, we are diving into the messy reality of refining those clusters into a coherent, actionable summary without burning through your entire API budget in one afternoon.
The Dimensionality Reduction Problem
Most embedding models produce vectors with massive dimensionality (we are talking 1,000+ dimensions). Humans can only visualize three. To understand if our clusters actually make sense, we use UMAP (Uniform Manifold Approximation and Projection). Think of it as squashing a high-dimensional warehouse plan into a 2D floor map while trying to keep similar items near each other.
Before you trust your clusters, you need to validate them. I’ve seen developers ship clustering logic that was essentially carving up data at random. You should always check your Silhouette Score. It measures how “comfortable” a chunk is in its assigned cluster versus the next best thing. A score of 0.056 might look low, but in real-world, messy documents like an employee handbook, some overlap is expected. It’s not about perfection; it’s about directional utility.
For more on managing complex data structures, check out my thoughts on LLM Agent Memory Architecture.
Strategic Chunking: Finding the Centroids
You have 1,360 chunks. You cannot summarize all of them individually—it’s too expensive and creates a second-order noise problem. Instead, we pick the most representative chunk from each cluster. In math terms, we find the centroid (the center point of the cluster) and then calculate the Euclidean distance for every chunk in that group.
The chunk closest to the center is your winner. It is the semantic “face” of that cluster. Here is how we implement that logic in Python:
import numpy as np
# Create a list to hold the best indices
closest_indices = []
# Loop through each cluster
for i in range(num_clusters):
# Calculate Euclidean distance from the cluster center
distances = np.linalg.norm(chunk_embeddings_array - kmeans.cluster_centers_[i], axis=1)
# Grab the index of the smallest distance
closest_index = np.argmin(distances)
closest_indices.append(closest_index)
# Sort them to keep the document flow logical
selected_indices = sorted(closest_indices)This approach reduced our context load from 220,035 tokens down to 4,219 tokens. That is a 98% reduction. That’s the difference between an API request that times out and one that ships in seconds.
The “Butterfingers” Handoff
Even with great clusters, the final “reduce” step is where things often break. In our experiment, the final summary leaned heavily into technical details (Kubernetes and demo systems) while thinning out the governance and pricing sections. This happened because the “reducer” LLM focused on the most detailed themes in the batch rather than the full spread.
To fix this, we don’t just need a better model; we need a better prompt. Your reduction prompt must explicitly require coverage across all thematic borough boundaries. If you’re struggling with this kind of context loss, you might want to read up on solving context rot with recursive models.
Look, if this LLM document summarization stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress and complex backend integrations since the 4.x days.
The Real-World Takeaway
Clustering isn’t magic; it’s a practical tool for survival in the era of massive token counts. Specifically, using UMAP for visualization and Centroids for selection gives you a scalable pipeline that actually works for enterprise-level documents. Don’t expect a single “reduce” step to be infallible. Always build in a topical-sanity check to ensure your final summary hasn’t accidentally ghosted the most important parts of your data.
For more technical details on clustering and visualization, I highly recommend the official UMAP documentation and exploring Silhouette Score analysis for your ML pipelines.
