Parsing JSON from the command line
I’m a big fan of command line shell utilities, and nowadays with the
proliferation of REST APIs and JSON documents, one of the commands
that I use more frequently in conjuction with curl is jq. jq
is a lightweight and flexible command-line JSON processor
and, we could say it is like awk or sed, but for JSON syntax. It may
installed via apt in Ubuntu, or downloaded directly from github. In
the following I’m going to show some illustrative use cases with jq:
Use case 1) Extracting information from Alfresco REST API
I recently wrote some shell scripts for extracting Alfresco sites information using jq.
#! /bin/bash
###
### getAlfrescoSites.sh
###
### Script for listing Alfresco Sites via REST API /service/api/sites
###
# Usage functions
usage() { echo "Usage: $0 [-f]" 1>&2; exit 1; }
# Command line options
while getopts "fu:p:h:" o; do
case "${o}" in
f)
FULL=1
;;
u)
MYUSER=${OPTARG}
;;
p)
MYPASS=${OPTARG}
;;
h)
ALFURL=${OPTARG}
;;
?)
echo "Invalid Option: -$OPTARG" 1>&2
exit 1
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
# Exports ALFURL,MYUSER,MYPASS
source ./exportENVARS.sh
FULL=${FULL:-0}
# Full/Brief report
if [ "${FULL}" == "1" ]; then
curl -s -u $MYUSER:$MYPASS "$ALFURL/service/api/sites" | jq -r '.[] | "(.shortName),(.visibility),(.title)"'
else
curl -s -u $MYUSER:$MYPASS "$ALFURL/service/api/sites" | jq '.[] .shortName' | sed -e 's#"##g'
fi
The first part of the script is for the command line options and it
really does not matter, just consider it as a template for another
Alfresco Shell Tools based on the REST API. The curl commands invoke
REST API, and jq is parsing the JSON output, extracting a
comma-separated file with site shortname, visibility and title (with
-f option).
$ ./getAlfrescoSites.sh -f
I use an complementary script for not to write passwords in every
command line (exportENVARS.sh)
#! /bin/bash export ALFURL=http://localhost:8080/alfresco export MYUSER=admin export MYPASS=secret
Use case 2) SOLR Admin Scripts.
In a SOLR cloud context, you may obtain the collections via the
following commands:
$ curl -s "http://localhost:8983/solr/admin/collections?action=CLUSTERSTATUS&wt=json" | jq ".cluster.collections" | jq '.[] .configName' $ curl -s "http://localhost:8983/solr/admin/collections?action=LIST&wt=json" | jq '.collections' [ "films", "techproducts", "gettingstarted", "zylk", "books", "exampledocs" ]
In Alfresco Search Services (with SOLR 6), you may easily look for
the numDocs,maxDoc and deletedDocs for each core with the following command:
$ curl -s "http://localhost:8983/solr/admin/cores?action=SUMMARY&wt=json" | jq '.Summary[].Searcher' | jq -r '"(.searcherName),(.numDocs),(.maxDoc),(.deletedDocs)"' Searcher@538b2748[alfresco] main,227555,252672,25117 Searcher@3bccbbfb[archive] main,78735,78748,13
or for checking SOLR nodes errors with:
$ curl -s "http://localhost:8983//solr/alfresco/afts?q=DOC_TYPE:ErrorNode&wt=json" | jq '.response.docs[].DBID'
Use case 3) Playing with my record collection database
at Discogs.
Finally in a more ludic way, we can play with Discogs REST API
to obtain the releases of an artist, for example, to obtain some John
Lennon releases in Discogs:
$ curl -s https://api.discogs.com/artists/46481/releases | jq '.releases[]' | jq -r '"(.id),(.year),(.title)"' 773968,1967,Comment j'ai gagné la guerre 1039108,1969,The KYA 1969 Peace Talk 5748043,1970,Reads "New Numbers" / Three Songs For Surrealists 8118946,1970,Love 72843,1970,Instant Karma! 72864,1970,John Lennon / Plastic Ono Band 72918,1970,Mother 515835,1970,The Short Rap 5822603,1971,Imagine / Oh My Love . .
or even, to do some export to CSV of my record collection at Discogs:
$ curl -s https://api.discogs.com/users/cesarista/collection/folders/0/releases --user-agent "FooBarApp/3.0"| jq '.releases[].basic_information' | jq -r '"(.id),(.year),(.title),(.artists[0].name),(.formats[0].name)"' 1945137,2009,Vacilando Territory Blues,J. Tillman,CD 1430463,2008,Fleet Foxes,Fleet Foxes,Vinyl 3706742,2012,Fear Fun,Father John Misty,Vinyl 1224530,1994,Crooked Rain Crooked Rain,Pavement,Vinyl 382617,1997,Barely Legal,The Hives,CD 9214979,2016,Veni Vidi Vicious,The Hives,Vinyl 3288249,2006,Pet Sounds (40th Anniversary Limited-Edition),The Beach Boys,CD 3247933,2011,Imagine - 40th Anniversary Special Edition,John Lennon,Vinyl 4882558,1966,Beach Boys' Party!,The Beach Boys,Vinyl 2616973,2009,The Beach Boys Today!,The Beach Boys,Vinyl . .
It is also very for transforming JSON, deleting some keys, and more.
Here, I compiled some of the examples:
Links:
