Using jq for parsing JSON documents

Parsing JSON from the command line

I'm a big fan of command line shell utilities, and nowadays with the proliferation of REST APIs and JSON documents, one of the commands that I use more frequently in conjuction with curl is jq. jq is a lightweight and flexible command-line JSON processor and, we could say it is like awk or sed, but for JSON syntax. It may installed via apt in Ubuntu, or downloaded directly from github. In the following I'm going to show some illustrative use cases with jq:


Use case 1) Extracting information from Alfresco REST API

I recently wrote some shell scripts for extracting Alfresco sites information using jq. 

#! /bin/bash
###
### getAlfrescoSites.sh
###
### Script for listing Alfresco Sites via REST API /service/api/sites 
###

# Usage functions
usage() { echo "Usage: $0 [-f]" 1>&2; exit 1; }

# Command line options
while getopts "fu:p:h:" o; do
    case "${o}" in
        f)
            FULL=1
            ;;
        u)
            MYUSER=${OPTARG}
            ;;
        p)
            MYPASS=${OPTARG}
            ;;
        h)
            ALFURL=${OPTARG}
            ;;
        \?)
            echo "Invalid Option: -$OPTARG" 1>&2
            exit 1
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Exports ALFURL,MYUSER,MYPASS 
source ./exportENVARS.sh

FULL=${FULL:-0}

# Full/Brief report
if [ "${FULL}" == "1" ]; then 
  curl -s -u $MYUSER:$MYPASS "$ALFURL/service/api/sites" | jq -r '.[] | "\(.shortName),\(.visibility),\(.title)"'
else
  curl -s -u $MYUSER:$MYPASS "$ALFURL/service/api/sites" | jq '.[] .shortName' | sed -e 's#\"##g' 
fi

The first part of the script is for the command line options and it really does not matter, just consider it as a template for another Alfresco Shell Tools based on the REST API. The curl commands invoke REST API,  and jq is parsing the JSON output, extracting a comma-separated file with site shortname, visibility and title (with -f option).

$ ./getAlfrescoSites.sh -f

I use an complementary script for not to write passwords in every command line (exportENVARS.sh)

#! /bin/bash
export ALFURL=http://localhost:8080/alfresco
export MYUSER=admin
export MYPASS=secret

 

Use case 2) SOLR Admin Scripts.

In a SOLR cloud context, you may obtain the collections via the following commands:

$ curl -s "http://localhost:8983/solr/admin/collections?action=CLUSTERSTATUS&wt=json" | jq ".cluster.collections" | jq '.[] .configName'

$ curl -s "http://localhost:8983/solr/admin/collections?action=LIST&wt=json" | jq '.collections'
[
  "films",
  "techproducts",
  "gettingstarted",
  "zylk",
  "books",
  "exampledocs"
]

In Alfresco Search Services (with SOLR 6), you may easily look for the numDocs,maxDoc and deletedDocs for each core with the following command:

$ curl -s "http://localhost:8983/solr/admin/cores?action=SUMMARY&wt=json" | jq '.Summary[].Searcher' | jq -r '"\(.searcherName),\(.numDocs),\(.maxDoc),\(.deletedDocs)"'
Searcher@538b2748[alfresco] main,227555,252672,25117
Searcher@3bccbbfb[archive] main,78735,78748,13

or for checking SOLR nodes errors with:

$ curl -s "http://localhost:8983//solr/alfresco/afts?q=DOC_TYPE:ErrorNode&wt=json" | jq '.response.docs[].DBID'

 

Use case 3) Playing with my record collection database at Discogs.
Finally in a more ludic way, we can play with Discogs REST API to obtain the releases of an artist, for example, to obtain some John Lennon releases in Discogs:

$ curl -s https://api.discogs.com/artists/46481/releases | jq '.releases[]' | jq -r '"\(.id),\(.year),\(.title)"'
773968,1967,Comment j'ai gagné la guerre
1039108,1969,The KYA 1969 Peace Talk
5748043,1970,Reads "New Numbers" / Three Songs For Surrealists
8118946,1970,Love
72843,1970,Instant Karma!
72864,1970,John Lennon / Plastic Ono Band
72918,1970,Mother
515835,1970,The Short Rap
5822603,1971,Imagine / Oh My Love
.
.

or even, to do some export to CSV of my record collection at Discogs:

$ curl -s https://api.discogs.com/users/cesarista/collection/folders/0/releases --user-agent "FooBarApp/3.0"| jq '.releases[].basic_information' | jq -r '"\(.id),\(.year),\(.title),\(.artists[0].name),\(.formats[0].name)"'
1945137,2009,Vacilando Territory Blues,J. Tillman,CD
1430463,2008,Fleet Foxes,Fleet Foxes,Vinyl
3706742,2012,Fear Fun,Father John Misty,Vinyl
1224530,1994,Crooked Rain Crooked Rain,Pavement,Vinyl
382617,1997,Barely Legal,The Hives,CD
9214979,2016,Veni Vidi Vicious,The Hives,Vinyl
3288249,2006,Pet Sounds (40th Anniversary Limited-Edition),The Beach Boys,CD
3247933,2011,Imagine - 40th Anniversary Special Edition,John Lennon,Vinyl
4882558,1966,Beach Boys' Party!,The Beach Boys,Vinyl
2616973,2009,The Beach Boys Today!,The Beach Boys,Vinyl
.
.

It is also very for transforming JSON, deleting some keys, and more.

 

Here, I compiled some of the examples:

Links:

00

More Blog Entries

thumbnail

0 Comments