Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been costly to incorporate into its homegrown solution. Engineers used API parity with the existing system to derisk the project and allow for a gradual and seamless migration.

Over the years, Netflix created a range of bespoke solutions for Titus, its container platform, including Compute Managed Batch (CMB) for managing and executing batch jobs on top of containerized compute. CMB managed capacity using tenant hierarchies and was able to federate workloads across multiple cells (Kubernetes clusters), leveraging Titus’ control plane APIs.

The company observed that since CMB’s creation in 2018, many of its key features have been incorporated into open-source projects that evolved within the Kubernetes ecosystem. Additionally, developing new features in CMB has become more difficult as the solution wasn’t as tightly integrated with Kubernetes. In the end, engineers decided to opt for Kueue as a replacement for CMB, due to its extensive capabilities and flexibility, wide adoption, and pace of innovation.

Kueue is a Kubernetes-native solution for batch job management and execution, offering a range of features, including job queuing based on priorities with different strategies, advanced resource management, multi-cluster and topology-aware scheduling, many integrations, comprehensive observability, and more.

High-level Kueue operation (Source: Kueue Documentation)

The team responsible for the migration to Kueue has opted to make the process completely transparent to existing CMB users and ensure that the new solution could support the required container launch rate and maximum throughput from the outset. The migration process was further tenant-bound and enabled easy rollback in case of issues.

Engineers mapped internal tenants in CMB to Cohors in Kueue and leaf tenants to a pair of ClusterQueue and LocalQueue resources. Resource flavors and nominal quotas were used to configure capacity requirements imported from CMB.

The migration from CMB to Kueue (Source: Netflix Technology Blog)

Netflix is currently managing millions of batch workloads in production using Kueue, but the migration is still underway. More recently, engineers were able to significantly improve average resource utilization by leveraging preemption-based fair sharing to maintain reservation semantics while lending idle capacity to other tenants.

The authors of the blog post shared lessons learned:

Don’t wait until the end to migrate the most complex use case. We decided early on to migrate our largest and most complex customer first. This allowed us to build confidence that we could later migrate other customers to Netflix Batch without issues, and resulted in the production migration lasting only 4 weeks.

Engineers highlighted that maintaining API parity while migrating underlying components derisked the project significantly and offered a seamless end-user experience. They also explained that running load tests in the non-prod environment helped them fine-tune performance-related configuration options to meet the platform’s throughput requirements.