Getting Ambari metrics via curl and Ambari REST API

Monitoring an Ambari cluster via curl command

During last days, I started to play with an Ambari cluster, based on the Hortonworks distribution. 
Ambari provides an agile interface for starting/stopping services in the cluster, modify configurations, and monitoring the different hosts and services of the cluster, providing alerts and metrics. Besides, Ambari provides a REST API for consuming the collected metrics and managing the cluster. This REST API is used in very interesting projects for monitoring a Big Data cluster (from the classical point of view) such as:

https://github.com/harisekhon/nagios-plugins

From the command line, we can curl the REST API services and parse the corresponding JSON results (this can be done in Linux with the help of jshon tool)

For example:

$ curl -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx?fields=alerts_summary 
{
  "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx?fields=alerts_summary",
  "Clusters" : {
    "cluster_name" : "lugxxx",
    "version" : "HDP-2.4"
  },
  "alerts_summary" : {
    "CRITICAL" : 0,
    "MAINTENANCE" : 0,
    "OK" : 110,
    "UNKNOWN" : 1,
    "WARNING" : 1
  }

where lugxxx is in this case be the name of our cluster.  

Note: In the upper comand, fields=alerts_summary is used for filtering the JSON info of the cluster. In fact, there is much more information inside if you ommit it

We may also take a look on the different services of the cluster:

$ curl --silent -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/services | grep service_name
        "service_name" : "AMBARI_METRICS"
        "service_name" : "FLINK"
        "service_name" : "HBASE"
        "service_name" : "HDFS"
        "service_name" : "HIVE"
        "service_name" : "KAFKA"
        "service_name" : "MAPREDUCE2"
        "service_name" : "PIG"
        "service_name" : "SLIDER"
        "service_name" : "SPARK"
        "service_name" : "TEZ"
        "service_name" : "YARN"
        "service_name" : "ZEPPELIN"
        "service_name" : "ZK-NIFI"
        "service_name" : "ZOOKEEPER"

And once, we identify the service, we may ask for one of them:

$ curl -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/services/HBASE?fields=ServiceInfo/state,ServiceInfo/maintenance_state
{
  "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx/services/HBASE?fields=ServiceInfo/state,ServiceInfo/maintenance_state",
  "ServiceInfo" : {
    "cluster_name" : "lugxxx",
    "maintenance_state" : "OFF",
    "service_name" : "HBASE",
    "state" : "STARTED"
  }
}

We may also see the different nodes of the cluster:

$ curl --silent -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/hosts| grep host_name
        "host_name" : "lug000.zylk.net"
        "host_name" : "lug001.zylk.net"
        "host_name" : "lug002.zylk.net"
        "host_name" : "lug003.zylk.net"
        "host_name" : "lug004.zylk.net"
        "host_name" : "lug005.zylk.net"
        "host_name" : "lug008.zylk.net"
        "host_name" : "lug066.zylk.net"

And also we may ask for what component is inside each node (lug000 in this case):

$ curl --silent -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/hosts/lug000.zylk.net| grep component_name
        "component_name" : "KAFKA_BROKER",
        "component_name" : "METRICS_MONITOR",
        "component_name" : "NODEMANAGER",
        "component_name" : "SECONDARY_NAMENODE",
        "component_name" : "ZOOKEEPER_SERVER"

Finally we may obtain some metrics for a given component of the node. For example, JVM metrics in NODEMANAGER:

$ curl -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/hosts/lug000.zylk.net/host_components/NODEMANAGER?fields=metrics/jvm
{
  "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx/hosts/lug000.zylk.net/host_components/NODEMANAGER?fields=metrics/jvm",
  "HostRoles" : {
    "cluster_name" : "lugxxx",
    "component_name" : "NODEMANAGER",
    "host_name" : "lug000.zylk.net"
  },
  "host" : {
    "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx/hosts/lug000.zylk.net"
  },
  "metrics" : {
    "jvm" : {
      "gcCount" : 2074,
      "gcTimeMillis" : 808166,
      "logError" : 0,
      "logFatal" : 0,
      "logInfo" : 179,
      "logWarn" : 10,
      "memHeapCommittedM" : 51.0,
      "memHeapUsedM" : 29.427994,
      "memNonHeapCommittedM" : 69.25,
      "memNonHeapUsedM" : 66.89617,
      "threadsBlocked" : 0,
      "threadsNew" : 0,
      "threadsRunnable" : 17,
      "threadsTerminated" : 0,
      "threadsTimedWaiting" : 37,
      "threadsWaiting" : 8,
      "JvmMetrics" : {
        "MemHeapMaxM" : 910.5,
        "MemMaxM" : 910.5,
        "MemNonHeapMaxM" : -1.0
      }
    }
  }
}

Finally a little tip, jshon command may help us to parse JSON from command line (or shell scripts) easily, and these commands may be the basis of a custom nagios/icinga shell or perl scripts for including email alerts in your Big Data stack. For example:

$ curl --silent -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/hosts/lug000.zylk.net/host_components/NODEMANAGER?fields=metrics/load/load_five
{
  "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx/hosts/lug000.zylk.net/host_components/NODEMANAGER?fields=metrics/load/load_five",
  "HostRoles" : {
    "cluster_name" : "lugxxx",
    "component_name" : "NODEMANAGER",
    "host_name" : "lug000.zylk.net"
  },
  "host" : {
    "href" : "http://lug066.zylk.net:8080/api/v1/clusters/lugxxx/hosts/lug000.zylk.net"
  },
  "metrics" : {
    "load" : {
      "load_five" : 0.42
    }
  }
}

$ curl --silent -u admin:secret http://ambari.lugxxx.zylk.net:8888/api/v1/clusters/lugxxx/hosts/lug000.zylk.net/host_components/NODEMANAGER | jshon -e metrics -e load -e load_five
0.420000

HINT: You can install jshon in Ubuntu 16.04 with:

$ sudo aptitude install jshon

Links:

• https://github.com/keenerd/jshon
• https://github.com/harisekhon/nagios-plugins