Introduction
See Part I to get an introduction to this series, and a guide on the Hard Way method.
In this post, we cover:
- What jq is
- Look at examples of it
- Introduce key terms
- Briefly look at its place in the software landscape
jq
This section introduces you to the jq command, starting with the simplest possible commands as well as a brief look at the most commonly-used flags.
What is jq?
jq is a program for parsing, querying, and manipulating JSON. More broadly, it’s a filter: it takes input and transforms it to output.
It can help you programmatically answer questions like:
- How many AWS VMs does my account have?
- How many of these VMs were created before last week?
- What is the running status of these older VMs?
- What is the list of VMs that have the tag ‘sales’, were created before last week, and are still running?
Before jq existed, the most common way to work out these things would be to use older command-line tools such as sed, awk, or programming languages like Python or perl. These could be highly sensitive to changes in input, difficult to maintain, prone to error, and slow. jq is much more elegant.
However, jq can be difficult for the uninitiated to understand and is more often used than understood. This book aims to help with that, to the point where you can confidently come up with your own solutions to challenges you face at work.
Invoking jq
Let’s get our hands dirty with jq.
Run this on your terminal:
$ echo '{}' | jqThis is the simplest JSON document you can pass to jq. The output of the echo command is piped into the jq command, which reads the output and ‘pretty-prints’ the JSON received for you because it’s going to the terminal rather than a file or another program. In this case, the input looks the same as the output, but the ‘pretty-printing’ involves emboldening the braces.
$ echo '{}' | jqInput to jq
jq takes a stream of JSON documents as input. This input can be via a pipe like this:
$ echo '{}' | jq
or from a given filename, like this:
$ echo '{}' > doc.json
$ jq . doc.jsonIt’s really important to understand that jq does not take a single JSON document as input (although it can). According to the documentation, it takes ‘a stream of JSON entities’.
What does this mean in practice? It means that if you run this:
$ echo '{}[]' | jqyou have just seen jq process ‘a stream of JSON entities’. What you inputted to jq was not valid JSON. If you want to prove this to yourself, just plug in {}[] to a JSON-parsing website (easily Google-able) or run this:
$ echo '{ {}[] }' | jqThe above just enclosed the ‘stream of JSON’ within a single JSON object (the surrounding {}), and jq throws an error, because what’s inside the single JSON object, ie:
{}[]is not itself a single valid JSON document.
Two jq Flags
The jq command has a number of flags which are useful to know about. At this stage of the book we’re just going to look at two, as to list them all would overwhelm you. These are the two I have seen used most often.
The -r Flag
The first is most often used in quick scripts and shell pipelines. All it does is remove the quotes from the items in the output. For example, just passing in a string to jq will output the string with the quotes surrounding it:
$ echo '"asd"' | jqAdding the -r flag removes the quotes:
$ echo '"asd" "fgh' | jq -rThe -S Flag
The second flag sorts fields in JSON objects. In this first example, the a and b items are output in the same order they were input:
$ echo '{"b":0,"a":0}' | jqAdding the -S flag sorts them by their key:
$ echo '{"b":0,"a":0}' | jq -SIn this simple example the value of sorting these fields is not so clear, but if you have huge JSON document to look through, then knowing this flag is a simple way to save lots of time hunting through screen after screen full of data.
What You Learned
- What JSON is
- What a JSON object is
- What a JSON array is
Exercises
1) Read the page https://www.json.org/json-en.html
2) Pick a programming language of your choice and parse a JSON document into it
Previous article
