Autoscaling vLLM and Distributed Embeddings on Open Infrastructure

Deploying generative AI workloads at production scale introduces unique infrastructure bottlenecks, from GPU memory fragmentation to uneven inference traffic. This session dives deep into operationalizing large language models using open cloud-native tools. We will explore how to architect distributed embedding pipelines and leverage vLLM inside containerized clusters for high-throughput inference. The discussion will cover concrete architectural strategies for cost optimization, monitoring GPU metrics with Prometheus, and setting up intelligent autoscaling rules tailored to handle volatile real-world AI traffic spikes smoothly.