Kibana dashboard for monitoring Liferay JVM via JMX

Monitoring JMX metrics with Metricbeat and Elastic Search

Some time ago we did a Kibana dashboard for JMX monitoring in Alfresco via Logstash. Today I’ll briefly let some notes for a similar configuration but with other tool of the Elastic Stack, Metricbeat, using a jolokia plugin for HTTP – JMX bridge. The main difference is that in the previous example, we had a logstash process for getting JMX information in a Tomcat node, and now we have a lighter agent, Metricbeat, which is ideally better suited for this purpose.

Besides the Metricbeat client, as con, we have to add / deploy jolokia.war under webapps directory in our Tomcat node, and to configure basic authentication and role for jolokia in the usual tomcat-users.xml file. We are using Elastic Search, MetricBeat and Logstash in version 7.2.0 and jolokia 1.6.2 (and Liferay 7.0 DXP deployed in Tomcat).

In tomcat-users.xml

  <role rolename="jolokia"/>
  <user username="monitor" password="secret" roles="jolokia"/>

For Metricbeat configuration (jolokia.jmx), we need to set JMX endpoint and the corresponding JMX beans to monitor (once the module is enabled):

- module: jolokia
  metricsets: ["jmx"]
  period: 30s
  hosts: ["localhost:8080"]
  namespace: "metrics"
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  username: "monitor"
  password: "secret"
  jmx.mappings:
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
    - mbean: 'java.lang:type=Threading'
      attributes:
        - attr: ThreadCount
          field: threading.thread_count
    - mbean: 'java.lang:type=OperatingSystem'
      attributes:
        - attr: SystemCpuLoad
          field: system.system_cpu_load
        - attr: SystemLoadAverage
          field: system.system_load_average
        - attr: OpenFileDescriptorCount
          field: system.opened_file_descriptors
    - mbean: 'Catalina:type=ThreadPool,name="ajp-nio-8009"'
      attributes:
        - attr: currentThreadCount
          field: threads.current_threads_count
        - attr: currentThreadsBusy
          field: threads.current_threads_busy
    - mbean: 'com.zaxxer.hikari:type=Pool (HikariPool-1)'
      attributes:
        - attr: TotalConnections
          field: pool.total_connections
        - attr: ActiveConnections
          field: pool.active_connections
  jmx.application:
  jmx.instance:
  fields:
    type: liferay-jmx
  fields_under_root: true

Then, we get JMX metrics via beats in logstash, having a custom index for JMX in Elastic Search:

input {
  beats {
    port => 5044    
  }
}

output {
  #stdout { codec => rubydebug }
  if [type] == "liferay-jmx" {
    elasticsearch {
      index => "metricbeat-jmx-%{+YYYY.MM.dd}"
      hosts => ["http://localhost:9200"]
    }
  }
}

Finally, we can create a custom dashboard representation of the obtained JMX data, for example JVM Used Heap or Java Threads, as shown below:

External links:

• https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-jolokia.html
• https://jolokia.org/download.html