In this blog I will discuss the role of protocol buffers in Mesos framework development. Protocol buffers are used extensively for messaging and serialization inside Mesos and when developing Mesos frameworks. I will show how to use its APIs when developing a framework using examples from the Mesos Elasticsearch project.
Like any distributed system, the components of a Mesos framework have to periodically send messages to each other. For instance, a scheduler sends a message to the executor telling it to use 2 CPUs and 1 GB of memory. Another example is an executor that sends back a 'TASK_RUNNING' status to the scheduler. Because message passing is so essential in building distributed systems, it is important to use a serialization system to program and deliver messages in a flexible manner. Mesos has chosen to use protocol buffers to solve this problem. There are many other serialization formats with trade-offs between certain features and performance characteristics.
Protocol buffers were created by Google back in 2008 for developers to define their own protocols and message formats. Protocol buffers generate code in C++, Java and Python. Mesos allows developers to create frameworks in several languages and use the generated protocol buffer APIs for that language. For instance, the Mesos Elasticsearch framework is written in Java and in it we use the generated Protos API classes. Let me show you an example of how to use these classes in the context of developing the Mesos Elasticsearch framework and afterwards I will show how protocol buffers work inside Mesos itself.
Checkout out this snippet from the Mesos Elasticsearch scheduler: the registered method. This method is called whenever a framework successfully registered with the Mesos master. It received several Protos objects, the frameworkID and the masterInfo. Inside the method we build up a list of resources and request them via the SchedulerDriver.
@Override
public void registered(SchedulerDriver driver, Protos.FrameworkID frameworkId, Protos.MasterInfo masterInfo) {
this.frameworkId = frameworkId;
try {
getState().setFrameworkId(frameworkId); // DCOS certification 02
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
LOGGER.info("Framework registered as " + frameworkId.getValue());
List resources = buildResources();
Protos.Request request = Protos.Request.newBuilder()
.addAllResources(resources)
.build();
List requests = Collections.singletonList(request);
driver.requestResources(requests);
}
List resources = buildResources();
Protos.Request request = Protos.Request.newBuilder()
.addAllResources(resources)
.build();
List requests = Collections.singletonList(request);
driver.requestResources(requests);
}
When you look at the request object you see a typical use of the protocol buffers APIs. It starts by creating a builder for one of the inner classes under the Protos class using newBuilder. After the resources have been added the build method is called to construct the request. Now let's look at how the resources are created. The buildResources method delegates to a helper class Resources which builds the resources. See the snippet below.
package org.apache.mesos.elasticsearch.common;
import org.apache.mesos.Protos;
/**
* Helper class for building Mesos resources.
*/
public class Resources {
public static Protos.Resource portRange(long beginPort, long endPort) {
Protos.Value.Range singlePortRange = Protos.Value.Range.newBuilder().setBegin(beginPort).setEnd(endPort).build();
return Protos.Resource.newBuilder()
.setName("ports")
.setType(Protos.Value.Type.RANGES)
.setRanges(Protos.Value.Ranges.newBuilder().addRange(singlePortRange))
.build();
}
public static Protos.Resource singlePortRange(long port) {
return portRange(port, port);
}
public static Protos.Resource cpus(double cpus) {
return Protos.Resource.newBuilder()
.setName("cpus")
.setType(Protos.Value.Type.SCALAR)
.setScalar(Protos.Value.Scalar.newBuilder().setValue(cpus).build())
.build();
}
public static Protos.Resource mem(double mem) {
return Protos.Resource.newBuilder()
.setName("mem")
.setType(Protos.Value.Type.SCALAR)
.setScalar(Protos.Value.Scalar.newBuilder().setValue(mem).build())
.build();
}
public static Protos.Resource disk(double disk) {
return Protos.Resource.newBuilder()
.setName("disk")
.setType(Protos.Value.Type.SCALAR)
.setScalar(Protos.Value.Scalar.newBuilder().setValue(disk).build())
.build();
}
}
This class has factory methods for creating CPU, disk, memory and port ranges resources. It uses the same builder pattern as described earlier. Note that many Protos types are complex nested structures. For instance the Scalar type is used inside the CPUs, disk and memory resources.
Generating protocol buffer code works by defining a .proto file and running the protoc compiler. Take a look at include/mesos.proto in the Mesos sources for the definition of the Resource type. Below is part of the definition of the Resource message. It contains several fields. A resource has a required type and it can contain different values: a scalar, ranges or a set. These fields are marked optional to support these different types. This is the 'union trick' that is mentioned in the comment. For more info see the protocol buffer techniques page from Google.
/**
* Describes a resource on a machine. A resource can take on one of
* three types: scalar (double), a list of finite and discrete ranges
* (e.g., [1-10, 20-30]), or a set of items. A resource is described
* using the standard protocol buffer "union" trick.
*
* TODO(benh): Add better support for "expected" resources (e.g.,
* cpus, memory, disk, network).
*/
message Resource {
required string name = 1;
required Value.Type type = 2;
optional Value.Scalar scalar = 3;
optional Value.Ranges ranges = 4;
optional Value.Set set = 5;
optional string role = 6 [default = "*"];
message ReservationInfo {
// Describes a dynamic reservation. A dynamic reservation is
// acquired by an operator via the '/reserve' HTTP endpoint or by
// a framework via the offer cycle by sending back an
// 'Offer::Operation::Reserve' message.
// NOTE: We currently do not allow frameworks with role "*" to
// make dynamic reservations.
// This field indicates the principal of the operator or framework
// that reserved this resource. It is used in conjunction with the
// "unreserve" ACL to determine whether the entity attempting to
// unreserve this resource is permitted to do so.
// NOTE: This field should match the FrameworkInfo.principal of
// the framework that reserved this resource.
required string principal = 1;
}
// More fields and messages...
}
The Resource type also contains nested messages. For brevity I show one them here: ReservationInfo. This message allows a framework to reserve resources by role and is currently being developed under MESOS-2018 - Dynamic reservation. The code is generated through Mesos' build process which is based on autotools. Makefile.am contains definitions for protocol files as well as the commands needed to generated the code.
Hopefully this blog gives you a good overview of using protocol buffers in the context of Mesos frameworks development. We would like to hear from your experiences with using Mesos, Mesos frameworks, Docker or related technologies. Also if you have any questions, a problem you want to solve, leave a comment below, talk to me on Twitter at @Frank_Scholten or contact us.