Learn jq the Hard Way,Part IV: Pipes

Other Posts

• Part I - JSON, Objects and Arrays
• Part II – The jq Command
• Part III – Filters

Pipes

The pipe is the next-most used feature of jq after filters. If you are already familiar with pipes in the shell, it’s a very similar concept.

How Important is this Post?

Pipes are fundamental to using jq, so this section is essential to understand. We will cover:

• The pipe (|)
• The ‘.[]‘ filter
• Nulls
• Array references

Setup

Create a folder to work in, and move into it:

$ mkdir ljqthw_pipes
$ cd ljqthw_pipes

The .[] Filter

Before we get to the pipe, we’re going to talk about another filter. This filter
doesn’t have a snappy name, so we’re going to refer to it as the .[] filter. It’s
technically known as the ‘array/object value iterator’.

Create a simple JSON document in a file:

$ echo '[{"username": "alice", "id": 1 }, {"username": "bob", "id": 2 }]' > doc1

This document is an array representing users on a system, with two objects in it.
Each object has a username and id name-value pair.

Since the file is a simple array, we can filter the individual objects by referring
to the array index in a fashion similar to other languages, by putting the index
number in square brackets:

$ jq '.[0]' doc1
$ jq '.[1]' doc1

If the referenced index doesn’t exist, then a null is returned:

$ jq '.[2]' doc1

You can also refer to negative indexes to reference the array from the end rather than the start:

$ jq '.[-1]' doc1
$ jq '.[-2]' doc1
$ jq '.[-3]' doc1

If you don’t provide an index, then all the items in the array are returned:

$ jq '.[]' doc1

Look at the output. How are the items returned?

• In a new array?
• In a series of JSON documents?
• In a comma-separated list?

Running these commands may help you decide. Note that in the below commands we are using a shell pipe, not a jq pipe. The two uses of the pipe (|) character are analogous, but not the same:

$ jq '.[]' doc1 | jq '.'
$ jq '.[]' doc1 | jq '.[0]'

It’s really important to understand that the .[] filter does not just work on
arrays, even though it uses an array’s square brackets. If a series of JSON
objects are supplied, then it will iterate through the name value pairs, returning
the values one by one. That’s why it’s called the ‘array/object value iterator’.

Type this out to see that in practice:

$ echo '{"username": "alice"} {"id": 1}' > doc2
$ jq '.' doc2
$ jq '.[]' doc2

This time you had no array, just a series of two JSON objects. The .[] filter
iterates over a series of objects, returning the values in it. Remember, it’s
formally known as the ‘array/object value iterator’.

Now try this. What happens? Why?

$ echo '"username"' > doc3
$ jq '.[]' doc3

This filter confused me for a long time when I was learning jq, as I thought the
.[] filter just removed the square brackets from the json, returning an array’s
contents, like this example does:

$ echo '[{"a": "b", "c": "d" }]' > doc4
$ jq '.' doc4
$ jq '.[]' doc4

While it obviously does do that in this case, you should now be aware that this
filter does more than just ‘remove the square brackets’!

Introducing The Pipe

OK, back to doc1. This time you’re going to run the .[] filter over it, and then
use a jq pipe to process it further. Run these commands and think about what
the pipe character (|) is doing.

$ jq '.' doc1
$ jq '.[]' doc1
$ jq '.[] | .["username"]' doc1

The first command shows you the JSON document pretty-printed, to remind you of the
raw content.

The second command (the ‘array/object value iterator’) iterates over the objects in
the array, and produces two JSON documents. This is an example of what was pointed
out above: the .[] filter does not simply ‘remove the square brackets’. If it
did, the resulting two objects would have commas between them. It turns the two
objects within the array into a series of JSON documents. Make sure you understand
this before continuing.

The third command introduces the jq pipe. Note that this time, the pipe character
(|) is inside the jq string’s single quotes, making it a jq pipe rather than
the ‘shell pipe’ we saw earlier. The jq pipe passes the output of the .[] operator
on to the next filter (.["username"]). This filter then effectively retrieves
the value of the username name/value pair from each JSON document from the output
of the first .[] filter.

You can run the last command in other ways too, with identical results:

$ jq '.[] | ."username"' doc1
$ jq '.[] | .username' doc1

jq will figure out what you mean if you use either of those shorthands.

Nulls

Not all JSON documents have objects in them with consistent names. Here you create
a document with inconsistent names in its objects (username and uname):

$ echo '[{"username": "alice"}, {"uname": "bob" }]' > doc4

Now see what happens when you reference one of the names in your query:

$ jq '.[] | .username' doc4

The object with the matching name outputs the its value, but the other
object outputs a null value, indicating that there was no match for
the query for that object.

Referencing the other object’s name swaps which value it outputs and which
results in a null:

$ jq '.[] | .uname' doc4

What You Learned

• What the .[] (array/object value iterator) filter does, and how it works on arrays and objects
• How to reference array items
• The various ways you can refer to names in JSON objects

Exercises

1) Create a file with three JSON documents in them: an array with a single name-value pair object in it, an array with two name-value pair objects in it, and a single bare object. Run it through the .[] filter and predict what output it will give.