Apache Mesos is an extremely valuable tool. One slight drawback, however, is that it can sometimes be hard to get started with custom frameworks. A ‘framework’ in Apache Mesos language is an ‘application’ that provides additional functionality. There are great benefits to writing frameworks, which we’ve spoke at length about here.

Mesos-Starter is an open source project that we use to quickly get started with new frameworks. It has already proven very useful for developing the Logstash Framework where thousands of lines of ‘boilerplate’ code that are necessary for frameworks was removed.

The key features of Mesos-Starter are:

• Filtering Mesos offers according to a set of rules
• Turn offers into Mesos tasks in either Docker or shell
• Reconciliation by persisting task state

The Road to Mesos-Starter

Last year we wanted to add a web interface to the ElasticSearch framework. My long background with Spring, and Spring Boot lately, led me in that direction. But only for the web interface. Everything else in the ES framework was left untouched.

Late last year I took over the development of the Logstash framework. The state of the framework was basically a proof of concept, which was a copy/paste of the ElasticSearch framework, with the only exception that dependencies and a few configuration bits were managed by Spring this time. Since I was the lead of the framework, I decided to go all in on Spring Boot.

I took some sweat, blood and tears to eventually get there, without really gaining an awful lot. The codebase was still very similar to the ElasticSearch Framework, but yet they're still two separate codebases. So that is why I decided to extract all code that looked similar into a Spring Boot Starter module because, as Phil puts it,

Boilerplate is what Spring is all about.

Now, with Mesos-Starter, Mesos developers avoid writing a lot of boilerplate code that is required to get started with writing Mesos frameworks.

The Goals of Mesos-Starter

In the spirit of Spring Boot, the starter project is very much convention over configuration. So to get started you basically just add the `spring-boot-mesos-starter` dependency to your project.

Of course it's not possible to make valid guesses about what the nature of a scheduler is supposed to do, so you will need *some* configuration, like giving your scheduler a name, it's minimum required resources for tasks, and finally what the task is.

Next to minimising the amount of boilerplate code, we also wants to help developers make some healthy, robust decisions on their frameworks. Mesos-Starter is very opinionated, especially on little things like how you de-register your framework. SIGKILL will just stop the scheduler and leave the tasks running, whereas a SIGTERM of the scheduler will make sure that the framework is properly de-registered from Mesos, so that Mesos can stop all running tasks.

Further to that, we have attempted to implement resiliency from different strategies. Due to the nature of the Logstash framework the best supported strategy is cluster wide system tasks. That means tasks that are running on every single host in the cluster.

There is support for running a number of instances, but effectively it can only scale up, and down in the case of natural death (which is actually not an anti-pattern). In the next release we will have support for changing the scale factor on the fly, either by your own algorithm or an external change.

The Future of Mesos-Starter

Up until now, the focus has been, and will remain, the Logstash framework. But developers from the Kibana and ElasticSearch frameworks have shown great interests in sharing and aligning with Mesos-Starter.

So within a foreseeable future we will have full support for at least these three kinds of applications (check the Mesos Starter readme for a full explanation):

• Cluster wide application deployments
• Stateless web application, and
• Distributed databases

Besides the different architectures we will add more features and we would really like to hear what features people would like to see. So please file an issue in our GitHub repository, or get in touch with me on Twitter, @mwldk.

Getting Started

To deploy a simple web application like Kibana you only need to write a Spring Configuration

@SpringApplication
public class KibanaScheduler {
  public static void main(String[] args) {
    SpringApplication.run(KibanaScheduler.class, args);
  }
}
 
  
  

And write a configuration file

spring:
  application:
    name: Kibana
  mesos:
    resources:
      distinctSlave: true
      scale: 3
      cpus: 0.1
      mem: 64
      port: 5601
  environment:
    ELASTICSEARCH_URL: http://elasticsearch.consul:9200
  docker:
    image: kibana
  
  

Hopefully you'll notice that all the magic is going on in the yaml configuration file where you hopefully will be able to configure all the boring stuff like filtering offers, mapping them into tasks and finally maintaining a number of running instances.

Demo

Below is a video demo showing the use of the mesos-starter project. It shows the entire process a user would follow to start a Mesos project, through to a running Mesos scheduler and executor, with zero lines of additional code. And all in about five minutes.

Summary

Spring Boot was really useful when it came to removing a lot of boilerplate code. We were able to use it to create a module that really makes it easy to get started with new Apache Mesos frameworks. Already we’ve removed thousands of lines of code from the Logstash framework. And as we build the tool out, we expect to remove more.