I’ve spent over 14 years dealing with production infrastructure, and most “production-ready” machine learning tutorials I’ve read are nothing of the sort. Building a Recommender System on Amazon EKS has less to do with the model you pick and more to do with the infrastructure glue, the race conditions in your pipelines, and the latency bottlenecks you don’t notice until you hit 500 requests per second.
Recently I was working through a multistage multimodal system for a large e-commerce platform. Besides tabular data, we had CLIP image embeddings and Sentence-BERT text features. When you try to scale a Recommender System on Amazon EKS like that, the usual “call the feature store on every request” approach will wreck your performance. Here’s how I build these so they hold up in production.
The multistage architecture: retrieval vs. ranking
You can’t score a million-item catalog on every request, so don’t try. We use a two-tower retrieval model to pull a few hundred candidates, then a heavier DLRM (Deep Learning Recommendation Model) ranker for the fine-grained ordering. We also keep Bloom filters in ElastiCache (Valkey/Redis) so users don’t get shown the item they clicked five minutes ago.
This is where it gets messy. In my experience the biggest bottleneck is the network overhead of fetching features, not GPU inference time. I’ve seen host memory bottlenecks halve performance on AWS just because the data wasn’t where it needed to be when the kernel asked for it.
Fixing the 195ms latency bottleneck
In a typical Recommender System on Amazon EKS setup with Feast or another feature store, your Python backend can spend around 200ms just waiting for item features. That’s far too slow. So I moved the item features into an in-process NumPy array cache inside the Triton Inference Server, which took lookup time from 195ms to effectively zero.
# bbioon_feature_cache.py
# A pragmatist's way to skip network calls during inference
import numpy as np
class ItemFeatureCache:
def __init__(self, feast_repo_path):
# We fetch all features ONCE during Triton initialization
self.store = np.load('all_item_features.npy')
print("Feature cache warmed up. Skip the network calls.")
def get_features(self, item_ids):
# O(1) lookup vs O(N) network call
return self.store[item_ids]After the switch to in-memory caching, throughput at concurrency=4 went up by more than 300%. If your item attributes don’t change much, there’s no reason to make a network trip to Redis for every candidate ID.
Scaling the Recommender System on Amazon EKS
Standard CPU-based autoscaling doesn’t work for ML workloads. Your CPU can sit at 20% while the GPU queue is completely backed up. So we run the Kubernetes Horizontal Pod Autoscaler (HPA) on a custom metric, Average Queue Time. If a request waits in the Triton queue for more than 30ms, Karpenter provisions a new GPU node (a p3.2xlarge or g4dn, say) within minutes.
Running these scaling clusters on AWS needs a strict MLOps pipeline. I use Kubeflow for the daily fine-tuning. We retrain the “Query Tower” (the user side) every 24 hours to keep up with changing preferences, without rebuilding the whole FAISS index for the item embeddings. It’s a bit of a “hack,” and it cuts our daily compute bill a lot.
If building a Recommender System on Amazon EKS is eating your dev hours, I can take it on. I’ve been working on WordPress and high-scale backend architecture since the 4.x days.
The practical takeaway
Don’t let the multimodal learning side distract you. A multimodal Recommender System on Amazon EKS succeeds or fails on data locality and on how you handle cold starts. Mask 5% of your user features during training so the model learns from context (device, time, location), and cache your static item features in memory every time. For more on high-performance serving, read the NVIDIA Triton Documentation and the EKS HPA Guide.
