Write your own Terraform provider: Part 1

This is the first part of a series of blog posts that explain how to write Terraform providers.

Before we start I would like to state that this article asumes a couple of things from you:

You have (some) experience with Terraform, the different provisioners and providers that come out of the box,
its configuration files, tfstate files, etc.
You are comfortable with the Go language and its code organization.

Because bootstrapping a Terraform provider can take some effort feel free to clone this Github repository to use it as your Terraform provider/plugin skeleton. It'll also help you go along with all the steps that we will mention later on.

Let's say that you want to write a Terraform provider for your awesome (cloud) provider. In practice, your Terraform configuration file would look like this:

 
provider "awesome" {
  api_key     = "securetoken=="
  endpoint  = "https://api.example.org/v1"
  timeout     = 60
  max_retries = 5
}
 
resource "awesome_machine" "speedy-server" {
  name = "speedracer"
  cpus = 4
  ram  = 16384
}

So, your provider called awesome supports four different fields:

api_key
endpoint
timeout
max_retries

You also want to have your own resource called machine (notice here that because of the way Terraform works your resource name is prefixed with the name of your provider, hence awesome_machine and not just machine) which supports the following fields:

name
cpus
ram

Where to start?

Start by calling plugin.Serve, passing along a "provider" function that returns a terraform.ResourceProvider.

 
func main() {
  opts := plugin.ServeOpts{
    ProviderFunc: Provider, // Read on to find the definition of this "Provider" function.
  }
  plugin.Serve(&opts)
}

Then define a function that returns an object that implements the terraform.ResourceProvider interface, specifically a schema.Provider:

 
func Provider() terraform.ResourceProvider {
  return &schema.Provider{
    Schema:        map[string]*schema.Schema{...},
    ResourcesMap:  map[string]*schema.Resource,
    ConfigureFunc: func(*schema.ResourceData) (interface{}, error) { ... },
  }
}

This schema.Provider struct has three fields:

Schema: List of all the fields for your provider to work. Things like access tokens, log levels, endpoints, region, etc.
The value of this field is a map[string]*schema.Schema, or in Spanish: a linked list where the key is a string and the value is a pointer to a schema.Schema.
A minimalistic example schema would look like this:
```
 
map[string]*schema.Schema{

  "api_key": &schema.Schema{

    Type:        schema.TypeString,

    Required:    true,

    Description: "Some short description here."

  }

}
```
Here we are saying that api_key is our configuration field in our configuration file; we are also specifying its type (schema.TypeString and not just string as this is required for Terraform to perform some validations when parsing the configuration file); we are also saying that is a required field: if the user does not specify a value for this field in the configuration file Terraform will throw an error and stop execution. Finally we add a short description to the field. There are more configuration options that can be specified for a schema field. You can see the complete list of fields of this struct here.
ResourcesMap: List of resources that you want to support in your Terraform configuration file. For example, if we were writing a Terraform provider for AWS and we wanted to support S3 buckets, Elastic Balancers and EC2 instances this is the place where you want to declare those resources.
The value for this field is a map[string]*schema.Resource, similar to the one of the Schema field, the difference being that this list points to schema.Resource. Let's take a look at one of the resources from the skeleton:
```
 
map[string]*schema.Resource{

  "awesome_machine": &schema.Resource{

    Schema: map[string]*schema.Schema{

      "name": &schema.Schema{

        Type:     schema.TypeString,

        Required: true,

      },

    },

    SchemaVersion: 1,

    Create:        func(d *schema.ResourceData, meta interface{}) {},

    Read:          func(d *schema.ResourceData, meta interface{}) {},

    Update:        func(d *schema.ResourceData, meta interface{}) {},

    Delete:        func(d *schema.ResourceData, meta interface{}) {},

  },

}
```
What we are doing here is until now pretty straight forward: we are declaring a list of resources. Each resource declaration has its own structure which is made out of a schema.Schema (we saw this already in the previous example when configuring the schema.Provider) and you probably also noticed that there are also a couple more fields like the SchemaVersion but I want to draw your attention specially towards the Create, Read, Update & Delete ones. These are the four operations that Terraform will perform over the resources of your infrastructure and they will be called according to the case for each resource. This means that if you are creating four resources the Create function will be called four times. The same applies for the rest of the cases.
The signature for these functions is func(*ResourceData, interface{}).
The ResourceData type will provide you with some goodies for getting the values from the configuration file:
- Get(key string): fetches the value for the given key. If the given key is not defined in the structure it will return nil. If the key has not been set in the configuration file then it will return the key's type's default value (0 for integers, "" for strings and so on).
- GetChange(key string): Returns the old and new value for the given key.
- HasChange(key string): Returns whether or not the given key has been changed.
- SetId(): Sets the id for the given resource. If set to blank then the resource will be marked for deletion.
It also offers a couple more methods (GetOk, Set, ConnInfo, SetPartial) that we won't cover on this post.

The second argument passed to the CRUD functions will be the value returned by your ConfigureFunc of your schema.Provider.
Following our example, meta in this case can be safely casted to our ExampleClient like this:
```
 
client := meta.(*ExampleClient)
```
Let's now take a look at the createFunc source:
```
func createFunc(d *schema.ResourceData, meta interface{}) error {
  client := meta.(*ExampleClient)
  machine := Machine{
    Name: d.Get("name").(string),
    CPUs: d.Get("cpus").(int),
    RAM:  d.Get("ram").(int),
  }
 
  err := client.CreateMachine(&machine)
  if err != nil {
    return err
  }
 
  d.SetId(machine.Id())
 
  return nil
}
```
As mentioned before we know that meta is indeed a pointer to our ExampleClient so we cast it. The client offers a CreateMachine method which receives a pointer to a Machine object, so we initialize that object populating its fields with the values that the user put in the configuration file using the Get method of the ResourceData that has been passed to our function. Then we perform the client.CreateMachine call, passing along the machine that we declared before. After that we check for errors and make an early return in case that something went wrong with the creation of the machine. Finally, if everything went fine we will make a call to SetId. This not only sets the resource ID in the tfstate file but also tells Terraform that the resource was successfully created.
For updating resources leverage the HasChange and GetChange functions. I will leave the implementation to your imagination and awesome software development capabilities.
It is important also to mention that if at any point you set your resource id to blank Terraform will understand that the resource no longer exists. This is convenient, for example, when you want to synchronize your remote state with your local state (when a resource has been removed remotely). This is a common task for the readFunc function.
ConfigureFunc: Make use of this function when you need to initialize some client with the credentials defined in the Schema part. You can find its signature here.

Any other example?

Check the skeleton project. I recommend you use it for when you're starting fresh with a new Terraform provider. Another good place to look for examples of complex use cases is the builtin providers that come along with Terraform.

Unit tests

When it comes to unit testing I suggest that you leave your Terraform provider as lightweight as possible. In the cases that we have worked on here at Container Solutions we have all the business logic in the client libraries (check for instance this Cobbler client library that we wrote) and has so far worked charms for us. Perhaps your use case is different. Perhaps not. Drop me a line either in the comments sections or on Twitter (@mongrelion). I would love to hear from you regarding this specific matter.

Final notes

This is not a full-grown Terraform provider. Far from it. But it will help you get started. Most of the documentation is in Terraform's source code which can be tricky at first to browse around. This is a small effort to gather some of the basic concepts to reduce the barrier and help other developers get started as quick as possible. And again, as stated in the beginning of this article, this is only the first part of a series of upcoming blog posts that will talk more about Terraform providers.

Some of the things that we want to talk about in the future are:

Partial state (or how to recover from faulty resource modification)
More complex schema definitions
How to run callbacks once all your resources have been created/updated/deleted

And possibly much more. Leave a comment if there is anything else that you would like us to cover on these series and thanks for reading!

Write your own Terraform provider: Part 1