Logging Apache Geode Client Queues
Introduction
Apache Geode provides a mechanism for client applications to subscribe to events of interest. This is done either through interest registration or continuous queries. In either case, once a client enables subscriptions and connects to the server members, a representation of that client called a CacheClientProxy is created on one or more server members depending on the subscription redundancy. The CacheClientProxy contains a queue of events of interest to its client. Events are added to the queue by the thread doing the operation (e.g. put). An asynchronous dispatcher thread takes events from the queue and dispatches them to the client. Sometimes it is necessary to know what events are contained in the queue for trouble-shooting purposes.
This article describes how to log the events contained in the CacheClientProxy’s queue.
Architecture
A CacheServerImpl instance is created when a server member starts. It creates an AcceptorImpl that accepts and handles new client connection requests. The AcceptorImpl creates a CacheClientNotifier which registers incoming subscription clients and creates and stores CacheClientProxy instances for them. Each CacheClientProxy creates a MessageDispatcher. The MessageDispatcher creates an HARegionQueue which in turn creates an HARegion which is an implementation of Region. The HARegion is the implementation of the queue. The MessageDispatcher takes events from the HARegionQueue and dispatches them to the client.
A simplified CacheClientProxy architecture is shown below:
Implementation
All source code described in this article as well as an example usage is available here.
The main object in this implementation is a Function called LogClientQueueEntriesFunction. It:
- gets the CacheServerImpl instance
- gets the AcceptorImpl instance from the CacheServerImpl
- gets the CacheClientNotifier instance from the AcceptorImpl
- gets the collection of CacheClientProxy instances
For each CacheClientProxy, it:
- gets its HARegionQueue
- gets the HARegion from the HARegionQueue
- gets the queue keys from the HARegion
- logs each queue key and value
In order to ensure that events are in the client queue when the LogClientQueueEntriesFunction is invoked, the example:
- configures the Region with a CacheListener called MessageDispatcherSlowStartInitializer which initializes a static boolean called CacheClientProxy.isSlowStartForTesting to true
- sets the java system property called slowStartTimeForTesting to 120000 (milliseconds)
The only purpose of these actions is to cause the MessageDispatcher to pause for 120 seconds before starting to process the queue. This should never done in an actual production system.
Get the CacheClientNotifier Instance
The getCacheClientNotifier method retrieves the CacheClientNotifier instance.
Log CacheClientProxy Queue Contents
The logClientQueues method gets all the CacheClientProxy instances and, for each one, logs its contents.
The logClientQueue method logs the contents of each CacheClientProxy’s HARegion queue.
Get Queue Keys
In addition to client event entries, the queue also contains an entry for each thread that puts events into the queue. That entry tracks the last dispatched event for that thread. The key for each of those entries is a ThreadIdentifier. The getQueueEntryKeys method filters and sorts the client keys (which are longs).
Log Queue Key and Value
The queue keys for events to be sent to the client are monotonically increasing longs. The queue values are generally HAEventWrapper instances which are light-weight representations of ClientMessage instances. A ClientMessage contains the actual data of the event and is stored in a map (called haContainer below). This allows multiple client queues in the same server to reference the same ClientMessage without duplicating the data. The logKeyAndValue method logs the long key and ClientMessage value of the queue entry.
Server Log Output
After running the LogClientQueueEntriesFunction, the server log files will contain a message like below.
Future
A gfsh command and Function that logs the client queue events like this example would be a useful addition to Apache Geode.
