MesosCon Europe 2017

Apache Mesos allows operators to run distributed applications across an entire datacenter and is attracting ever increasing interest. As much as distributed applications see increased use enabled by Mesos, Mesos also sees increasing use due to a growing ecosystem of well integrated applications. One of the latest additions to the Mesos family is Apache Flink.

Flink is one of the most popular open source systems for real-time high scale data processing and allows users to deal with low-latency streaming analytical workloads on Mesos.

In this talk we explain the challenges solved while integrating Flink with Mesos, including how Flink’s distributed architecture can be modeled as a Mesos framework, and how Flink was integrated with Fenzo. Next, we describe how Flink was packaged to easily run on DC/OS.

Running distributed TensorFlow is challenging, especially if you want to train large models on your own infrastructure. In this talk, Kevin Klues and Sam Pringle will present an open source TensorFlow framework for distributed training on DC/OS. This framework addresses several challenges associated with distributed TensorFlow, and they hope it will make life much easier for anyone doing machine learning with large models/datasets. Kevin will introduce TensorFlow on Mesos and DC/OS, and Sam will give a live demo of the framework.

Is there a way to combine new architectural patterns such as micro-services with Big Data technologies and run everything in Mesos?

In this talk I will present a novel, highly scalable Machine Learning platform for our Machine Translation use-cases.

I will explain how, in order to reach this goal, we have combined a wide variety of Big Data technologies(like Kafka, HBase, Hadoop), and I will discuss the challenges that we have faced along the way. I will also present how we adopted a containerized micro-services architecture(based on Mesos, Docker, Zookeeper) in order deploy our highly scalable Machine Learning platform.

In this session, we will illustrate how to develop, deploy and manage FAST data solutions at scale on DC/OS and Azure. We will discuss the challenges of stateful containers in the cloud and provide guidance on how to implement both Cassandra and Kafka in DC/OS. We will further discuss how to manage your deployed solution with a partner solution. This session will be demo heavy, illustrating how to develop Cassandra and Kafka applications locally, run them at scale in DC/OS and manage them with a 3rd party workflow solution.

Organizations Mesos and Apache Spark together to gain insight from large amounts of data. It is common for Spark to process data stored in disparate public cloud storage, such as Amazon S3, Microsoft Azure Blob Storage, or Google Cloud Storage as well as on-premise data on HDFS, Ceph or ECS. This architecture results in sub-optimal performance as data and compute are not co-located.

Using Alluxio, a memory speed virtual distributed storage system, deployed on Mesos enables connecting any compute framework, such as Apache Spark, to storage systems via a unified namespace. Alluxio enables applications to interact with any data at memory speed. Alluxio can eliminate the pains of ETL and data duplication, and enable new workloads across all data. Gene will discuss the architecture of Mesos, Spark and Alluxio to achieve an optimal architecture for enterprises.

Most modern distributed applications like Cassandra and HDFS provide replication of data across nodes and failure zones to be able to deal with failures. But the time taken to recover to a pre-failure level of redundancy in cases of permanent node failures can be large, since a lot of data needs to be copied over to the new node. Also, some of these applications cannot accept new writes on the nodes being bootstrapped, further increasing the recovery time.

Dinesh Israni will talk about how you can use dcos-commons frameworks for Cassandra, Elasticsearch, HDFS, Kafka and Spark along with External Persistent volumes to reduce recovery times for your distributed applications and achieve high availability for applications that don’t provide replication.

As a systems integrator specializing in cloud transformation projects, New Context has helped customers run mission-critical applications on DCOS. As part of that work, we’ve overcome a host of challenges that pop up when running stateful services like databases, queues, and key-value stores on top of DC/OS.

DC/OS supports local and external volumes for stateful applications, but there are a number of documented “caveats” that must be overcome like volumes being pinned to hosts, inability to dynamically provision volumes at run time, resource requirements being fixed at task launch, and being limited to one task per volume.

This talk will provide background on the main gotchas of running stateful services on Marathon and DCOS, and will discuss how to overcome them based on real-world projects conducted alongside some of the largest container users in the world.