MesosCon Europe 2017

At Yelp, we all strive to increase developer productivity by decreasing the time to test, deploy, and monitor changes. To enable developers to push code safely, we run more than 20 million tests every day. Yelp has a monolith web application which has 100,000 tests; running them sequentially takes approximately 2 days to finish. We built an in-house distributed system called Seagull, which splits these 100,000 tests into smaller chunks(bundles) using our bespoke algorithm and runs these chunks in parallel on the compute cluster to finish all the tests in less than 10 mins. Seagull uses Apache Mesos to schedule these run test bundles on AWS spotfleet. We have written a custom autoscaler for AWS spotfleet which dynamically adjusts the cluster capacity based on different utilization metrics for optimal use of resources. Seagull runs more than 2 million ephemeral docker containers every day.