I have been working on a Go project for the past couple of months, I have found the language pleasant to use and the tooling has been pretty impressive. I am not going to talk about this though, I am more interested in how to build out a 12 Factor app in Golang.
I won’t go into detail in this post on the 12 factors. But if you would like to know more the following (https://12factor.net/) provides a decent ideological overview of the principals. I will be focusing on just one factor, configuration management.
Historically, configuration management has been an ad-hoc experience. Every language and framework has its own idiosyncratic ways to configure applications, different configuration file-formats are developed, informal rules are partially enforced. Generally, telling a program what to do is a game of reading the documentation.
I will justify myself with a few examples. I will stick to Golang projects which have reasonable adoption. The same can be said of most programming languages though. Traefik uses TOML for configuration, a specific format that maps unambiguously to a Map/Dictionary/Hash. Caddy uses a custom configuration syntax similar in style to the nginx format and Docker uses a json file (or uses command-line switches).
This isn’t really a problem when you run 5-6 instances on a single machine, you can upload the configuration file, install the package and start up the tool. Before I get into why this becomes a problem at scale, I will first jump into what constitutes a good configuration management system.
In essence, what one needs in a configuration management system is fine-grained control of the configuration parameters in each environment without having to alter the codebase between deploys. What this means is not grouping a set of configuration parameters together under a single environment. With this configuration model you will need a ‘development’, ‘testing’ and 'production' environment. This doesn't cover the case of wanting to deploy in multiple different production environments (i.e. on-premises/cloud).
This doesn’t scale effectively, information is repeated, prone to error and requires updating the codebase for each configuration change. Storing a config file with your project isn’t very effective either as now you need to figure out how to embed and/or transport the config file along with the code.
The 12 Factor app method recommends setting all configuration options within Environment variables, this allows embedding the configuration within the environment that the application will be running. It is also language agnostic and provides a generic form of configuration while enforcing a strict separation between code and configuration.
When I started this golang project, I had a look through some of the other projects that were around so that I could find standard tooling for configuring the system correctly. I came across Cobra, and subsequently Viper as the current Industry standard used in Kubernetes, Rkt and other containerisation technologies.
Cobra describes itself as 'A Commander for modern Go CLI interactions'. It facilitates and abstracts away the handling and parsing of commandline switches and wraps it all up in an easy to use interface. It is the tool used to create the 'command sub-command' CLI applications popular in the container industry.
Viper is a companion library to Cobra which allows configuration through environment variables.
The documentation for Cobra is very good so writing out the CLI was a breeze. Writing a simple hello world example is shown below:
package main
import (
"os"
"fmt"
"github.com/spf13/cobra"
)
var rootCmd = &cobra.Command{
Use: "main",
Short: "CobraDemo",
Long: `Demo Program`,
}
var (
greeting string
subject string
)
func init() {
helloCmd := &cobra.Command{
Use: "hello",
Short: "Say hello",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println(greeting, subject)
},
}
helloCmd.Flags().StringVar(&subject, "subject", "World", "to whom do you say it?")
rootCmd.AddCommand(helloCmd)
}
func main() {
rootCmd.PersistentFlags().StringVar(&greeting, "greeting", "Hello", "what say you?")
if err := rootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
We can run this relatively easily as such:
go run main.go hello --subject=You
(out) Hello You
This is a simple way to get started with passing in configuration flags and is what I used for the first few weeks of the project. As soon as I started trying to deploy the application in Kubernetes though I started having problems, how do I inject the configuration values without statically defining them in the Deployment manifest file? I needed a way to inject the configuration data separately from the program itself. The answer, configMaps. A configMap is essentially a yaml file which can be injected into the environment of a pod. This is very useful for keeping fine-grained control of our configuration while also keeping it strictly separate from the codebase.
I had a problem though, how would I read in all of the variables while allowing me to override them from the command-line if needed, all without making a huge mess. This is where Viper comes in.
package main
import (
"os"
"fmt"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var rootCmd = &cobra.Command{
Use: "main",
Short: "CobraDemo",
Long: `Demo Program`,
}
func init() {
helloCmd := &cobra.Command{
Use: "hello",
Short: "Say hello",
Run: func(cmd *cobra.Command, args []string) {
greeting := viper.GetString("greeting")
subject := viper.GetString("subject")
fmt.Println(greeting, subject)
},
}
helloCmd.Flags().String("subject", "World", "to whom do you say it?")
viper.BindPFlag("subject", helloCmd.Flags().Lookup("subject"))
viper.SetDefault("subject", "World")
rootCmd.AddCommand(helloCmd)
}
func main() {
viper.SetEnvPrefix("demo") // Set the environment prefix to DEMO_*
viper.AutomaticEnv() // Automatically search for environment variables
rootCmd.PersistentFlags().String("greeting", "", "what say you?")
viper.BindPFlag("greeting", rootCmd.PersistentFlags().Lookup("greeting"))
viper.SetDefault("greeting", "Hello")
if err := rootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
This can be run in several different ways:
go run main.go hello --greeting=Goodbye
(out) Goodbye World
DEMO_SUBJECT=You go run main.go hello --greeting=Goodbye
(out) Goodbye You
One can see that each configuration option needs to be annotated to specify that it is configurable either a CLI switch or an environment variable. This allows for a clean interactive UX when the tool is used manually, while simultaneously allowing complete customisation for when the tool is run within a Highly-Available environment.
There are a few key takeaways from this code snippet, one is that the environment variables are contained in a namespace. This means one won’t accidentally clobber someone else's configuration. The second is that we want to keep configuration as malleable as possible, allow for setting a base override, and allowing developers to set custom configurations for their own machine or development requirements.
Configuration is an essential part of any product, it needs to be thought about throughout the building process of the application. This is especially true when building for the cloud.
Looking for a new challenge? We're hiring!