Advice on LLM document summarization has settled on “just chunk it and hope for the best.” Try that on a 200,000-word handbook and you get hallucinations and context drift. Fourteen years of development has taught me that brute force rarely scales. When the context window is the bottleneck, the fix is a better filter, not a bigger request.
Part 1 covered the “Lost in the Middle” problem and used K-means to group document chunks into semantic folders. This part is about refining those clusters into a summary you can act on, without spending the whole API budget in one afternoon.
The dimensionality reduction problem
Most embedding models produce vectors with 1,000 or more dimensions. People can visualize three. To check whether the clusters make sense, we use UMAP (Uniform Manifold Approximation and Projection), which squashes a high-dimensional layout into a 2D map while trying to keep similar items near each other.
Validate the clusters before you trust them. I have seen clustering logic ship that was carving up data more or less at random. Check the Silhouette Score, which measures how comfortable a chunk is in its assigned cluster compared with the next best one. A score of 0.056 looks bad on paper, but messy documents like an employee handbook overlap by nature, so treat the number as a direction rather than a grade.
My write-up on LLM Agent Memory Architecture goes further into handling complex data structures.
Finding the centroids
You have 1,360 chunks. Summarizing each one is too expensive, and it creates a second-order noise problem on top of the cost. Instead we take the most representative chunk from each cluster: find the centroid (the center point of the cluster), then calculate the Euclidean distance for every chunk in that group.
The chunk closest to the center wins. It stands in for the whole cluster. In Python:
import numpy as np
# Create a list to hold the best indices
closest_indices = []
# Loop through each cluster
for i in range(num_clusters):
# Calculate Euclidean distance from the cluster center
distances = np.linalg.norm(chunk_embeddings_array - kmeans.cluster_centers_[i], axis=1)
# Grab the index of the smallest distance
closest_index = np.argmin(distances)
closest_indices.append(closest_index)
# Sort them to keep the document flow logical
selected_indices = sorted(closest_indices)That took the context load from 220,035 tokens to 4,219, a 98% reduction. In practice it is the difference between a request that times out and one that comes back in seconds.
Where the reduce step drops topics
Even with good clusters, the final reduce step is where things break. In our run, the summary leaned hard into technical detail (Kubernetes and demo systems) and thinned out the governance and pricing sections. The reducer LLM had followed the most detailed themes in the batch instead of covering the spread.
The fix lives in the prompt rather than the model. Your reduction prompt has to require coverage across every thematic boundary you clustered on. If context loss is the problem you keep hitting, my post on solving context rot with recursive models is related.
If this LLM document summarization work is eating your dev hours, I can take it on. I have been doing WordPress and complex backend integrations since the 4.x days.
What holds up in practice
Clustering is a practical way to keep token counts manageable. UMAP for visualization and centroids for selection gives you a pipeline that survives enterprise-sized documents. Do not treat a single reduce step as reliable. Add a topical sanity check so you notice when the final summary has quietly dropped the parts you needed most.
The official UMAP documentation covers clustering and visualization in more depth, and Silhouette Score analysis is worth reading before you wire this into an ML pipeline.
