Implementing a Spring Boot JMX Notification Listener for Apache Geode

Introduction

Apache Geode issues JMX Notifications for specific system events and system alerts (e.g. warning and severe messages).

JMX Notifications include but are not limited to:

  • when members join and leave the DistributedSystem
  • when Regions are created and destroyed
  • when GatewaySenders are created, started and stopped
  • when warning or above messages are logged (depending on the DistributedSystemMXBean alert level)

This article describes a Spring Boot JMX Notification listener application that connects to and listens for JMX Notifications from Apache Geode members.

Implementation

All source code described in this article is available here.

The GeodeJmxNotificationListener:

  • connects to the MBeanServer running in the JMX manager (the locator)
  • changes the DistributedSystemMXBean’s alert level from severe (the default) to warning
  • adds itself as a NotificationListener to the DistributedSystemMXBean
  • handles JMX Notifications by handing them off to a JmxNotificationHandler for processing
  • implements the JmxConnectionHandler interface to manipulate the JMX connection
  • implements the JmxNotificationsAccessor interface to provide access to the JMX Notifications.

The JmxNotificationClient application class autowires the GeodeJmxNotificationListener as a JmxConnectionHandler, and its connectJmxConnectionHandler Bean method causes the connection to the JMX manager to be made.

The GeodeJmxNotificationController REST controller class autowires the GeodeJmxNotificationListener as a JmxNotificationsAccessor to access the JMX Notifications.

Connect to the MBeanServer

The connectToMBeanServer method repeatedly attempts to connect to the MBeanServer in the JMX manager until it succeeds. A connection attempt will be made every 5 seconds by default (controlled by the jmx.listener.connection.retry.delay property).

The attemptToConnectToMBeanServer method actually makes the connection like:

If the connection cannot be made, messages like these will be logged:

Once the connection is made, messages like these will be logged:

Change Alert Level

The changeAlertLevel method invokes the the changeAlertLevel operation on the DistributedSystemMXBean to change the alert level from severe to warning like:

Add NotificationListener

The addNotificationListener method adds the GeodeJmxNotificationListener as a NotificationListener to the DistributedSystemMXBean like:

Handle Notifications

The handleNotification method handles JMX Notifications by handing them off to each JMXNotificationHandler like:

Check Connection to the MBeanServer

The checkConnection method verifies the connection to the MBeanServer in the JMX manager is still valid. This method is scheduled to run every 1 second by default (controlled by the jmx.listener.check.connection.delay property).

If the connection is lost, output like this will be logged and the connection will be re-attempted using the connectToMBeanServer method described above:

Logging

Implementations of the JmxNotificationHandler interface do the actual handling of the JMX Notification. The example provides one implementation called LoggingJmxNotificationHandler which just logs the JMX Notification. Other implementations could post the JMX Notification to Slack, email it or store it in an external database, for example.

Examples of logged Notifications include:

DiskStore disk usage warning:

GatewaySender remote locator warning:

Member joined:

Member departed / crashed:

Region created:

REST APIs

The GeodeJmxNotificationController provides several REST APIs for accessing the JMX Notifications including:

  • getting the JMX Notifications as JSON
  • clearing the JMX Notifications

Getting the JMX Notifications

Each JMX Notification is added to a list as it is received. The list of JMX Notifications can be retrieved using the getnotifications request method like:

curl http://localhost:8080/getnotifications

The JMX Notifications are returned as JSON strings. Some examples are:

DiskStore disk usage warning:

GatewaySender remote locator warning:

Member joined:

Member departed / crashed:

Region created:

Clearing the JMX Notifications

The list of JMX Notifications can be cleared using the clearnotifications request method like:

curl -X POST http://localhost:8080/clearnotifications

Note: the list of JMX Notifications is cleared automatically every 24 hours by default (controlled by the jmx.listener.clear.notifications.delay property) using the scheduled clearNotifications method.

Future

An out-of-the-box Spring Boot JMX Notification client with various supported JmxNotificationHandler plugins would be a useful addition to Apache Geode.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store