This is a write up of the excellent webinar on securing microservices given by Sam Newman, author of “Building Microservices” last week. The webinar was a teaser for a new 2-day course from Sam and Adrian Mouat (author of “Using Docker”) and Container Solutions that debuts this month in Amsterdam. Pretty much all the clever observations in this post I pre-attribute to Sam.
The first thing that struck me in Sam’s webinar was that I’d written a whole article on Microservices architecture a few weeks ago but I hadn’t once mentioned security. That wasn’t because I don’t care about security or it’s an innate mystery to me, it’s just that it didn’t strike me as a big issue to talk about. How wrong I was! Of course it is! And Sam very succinctly told me why.
As Sam put it, microservices are independently deployable processes. That means in a system of microservices you can start, stop or replace any of them at any time without breaking everything. That’s great for reducing clashes between developers, increasing resilience and improving feature velocity, but for security it’s a double edged sword. It can enable you to make everything more secure with better defense in depth (hurray!) but if you don’t make a significant effort it can leave you in a much more exposed position than a monolith (damn!).
Security-wise the good thing about microservices is that by dividing your system up you can separate data and processes into “highly sensitive or critical” and “less sensitive” groups and put more energy, focus and expenditure into protecting your high sensitivity and critical stuff. In the olden days of a monolith, everything was together in one place so it all had to be highly protected (or not, as the case may be). Your eggs were all in one basket, which colloquially we tend to disapprove of (although it is not actually an unknown security strategy - WW2 convoys very successfully made use of a heavily defended single basket).
Microservices give you more opportunity to layer your defenses (defense in depth) but also more opportunities to fail to do so.
I’m sure you’re getting the picture that this advantage isn’t entirely clear cut.
The downside of microservices is that by spreading your system out over multiple containers and machines you increase the attack surface. You have more to protect.
What kind of attack surfaces are we talking about?
Basically, microservices are very powerful but also hard. They can improve your security but without careful thought they will probably reduce it. According to Sam, with microservices security needs to be planned in from the start.
Sam will go into this in more detail in his course but basically he recommended we use threat modeling.
One useful technique for threat modelling is thinking up “attack trees” that cover every way a baddie could possibly attack your system and then putting a cost/difficulty against each attack.
For example: breaking into my house. The lowest attacker-cost way in would be climbing through an open window while I was out (easy). The highest cost way in might be fighting the sabre-toothed tiger on my doorstep (hard).
The idea is not to make every attack impossible but to make every attack too costly. Apparently my sabre-toothed tiger was complete overkill, I should just remember to close my windows.
Some attacks are physical (breaking a window) and some are social (persuading me to let you in to read a meter). The first you battle with tools, the second with processes.
According to Sam, a useful way to think about security and how to handle the attack points you’ve just uncovered with your attack tree is as a 4 step process:
So, what tools does he say we have that can secure microservices?
The first and easiest is HTTPS. If any of your microservices communicate over HTTP then stop. Move them to HTTPS. Just because a connection is inside your system perimeter that doesn’t mean we can assume it’s safe from snooping. The good news is HTTPS is not as hard as it used to be. There are now great tools and free certificates from LetsEncrypt, amongst others. HTTPS also doesn’t slow things down anymore because most servers are optimized for encryption.
Using HTTPS verifies the data hasn’t been tampered with and verifies the callee, but it doesn’t verify the caller. For that you’ll need some form of client-side auth, such as client-side certificates. Don’t have a heart attack, those are also easier than they used to be. Sam says take a look at Lemur from Netflix.
If you are using other forms of communication rather than REST/HTTP then there are ways to secure that too but Sam saved the details for his course.
Certs covers service-to-service authentication but what about user auth? You still need to use OAuth or equivalent to cover that. You'll also have to consider whether or not services further downstream need to revalidate what a logged in user can do. The course will apparently tell you more about that.
You’ll probably also want to use SDN/network security and policy enforcement like that from ProjectCalico or Weave to make sure that traffic only ever comes at your services from services they are allowed to talk to. Defense in depth folks! Policy AND encryption!
Everyone’s security “open window”, however, is usually patching. You’ve got to keep all your machines and containers patched for vulnerabilities. In a microservice environment you are probably going to end up with too many units to do this manually. You’ll quickly need to automate this process. Look at tools that can help you do so.
Microservices lend themselves to a best-of-breed or polyglot approach where everyone runs their dream stack. That has security advantages and disadvantages. Commonality is easier to secure until you’ve got your head round everything and automated loads of it. Keeping 5 stacks secure and patched is easier than 500. The benefit of diversity, however, is if your hackers do find an exploit then maybe they can take it less far, just compromise one microservice. Pros and cons abound but Sam recommended that you start with a smaller number of stacks and patch them carefully.
Logs! And keep your logs for a very long time. Sam points out that the usual demand for logs is from dev diagnosing field issues maybe a few weeks ago. Intrusion detection may be investigating problems from a long time earlier than that so you'll need to keep logs longer. Look at the ELK stack: Logstash, Elasticsearch and Kibana for example.
IP-based security appliances/tools that detect unusual behaviour inside or at your perimeter are also very useful.
Responding is less about tools and more about processes. Knowing what to do and doing it.
1. Don’t panic!
2. Don’t ignore it!
Have processes pre-defined, carefully thought-through, and tested for acting on attack detection. Don’t wait until it happens because in the heat of the moment you’ll make mistakes.
This is the bread and butter stuff. The recovery is actually just best practice for recovering from any disaster:
So, Sam’s overall conclusion was Microservices are a hugely powerful tool for letting you build defense in depth, but they also give you loads more opportunities to screw up and leave a window open so you need to think and plan.
I suspect the general advice is Don’t Panic. but Don’t Ignore it!
If you want to understand microservices security I’d strongly recommend you send someone to one of Sam and Adrian’s courses in Amsterdam or London. Book now - unsurprisingly they’re filling up rapidly.
In the meantime, Adrian Mouat will be giving another webinar on securing containers next week so make sure you book that too so you don't forget to listen.
More about the Secure Microsevices Course By Sam Newman and Adrian Mouat in Amsterdam and London
Read more about our work in The Cloud Native Attitude.
