Black Boxes: Monitoring Solr (JMX Edition) | AppNeta

Black Boxes: Monitoring Solr (JMX Edition) | AppNeta

Solr exposes hundreds of JMX metrics across dozens of categories, and efficient use of them can help you delve into Solr performance in a variety of ways. Some metrics are better for providing a high-level view of Solr’s overall workflow. The queryResultCache category, pictured above, provides a snapshot of how often your data was successfully cached, as well as how often cache entries had to be evicted due to insufficient space. Other metric categories are more granular and provide detail at the level of classes, or even objects. An update request will be routed to a different handler depending on whether the data was provided in XML, CSV, or JSON; each of these update handlers exposes metrics independently, like how long it has been running and the number of errors.

JMX metrics can even provide insight into advanced Solr use cases, like modifying result scoring to permit n-dimensional spatial searches or customizing results based on user data stored in Redis. Even without addingcustom JMX metrics, Solr will report enough data to allow you to separately track the effectiveness of these custom searches relative to more traditional queries.

After checking the metrics for that node’s active Searcher instance, you realize you didn’t set up Solr to warm the cache – it was starting off empty! Now you know to make a quick configuration change next time you spin up an instance so that the first users routed to it will have acceptable performance.

Purpose-built JMX monitoring tools like jconsole are great for browsing the available metrics to see what’s available, but they’re horrible for pulling out the ones you want in a hurry. They also allow ‘write’ operations like initiating garbage collection or clearing caches – definitely not something you want to give out to every developer!

On a day to day basis, it’s more common to read JMX metrics via automated, ‘read-only’ monitoring tools likeNagios, Ganglia, or AppNeta TraceView. These tools not only present a number of metrics at once, but they also generally let you filter down to a meaningful subset of the hundreds of lines exposed by Solr. On the other hand, “health check”-style metrics aren’t necessarily the only way to look the problem. Each request has a number of metrics it can generate, and bringing together these data sources in one application has some real advantages. Looking at an individual request can tell you exactly what went wrong, it’s often the context of JMX data that says why. Examining the concurrent host activity can disambiguate between whether a pause was due to a garbage collection event in the JVM or an overloaded document cache in Solr forcing additional disk access.

Read full article from Black Boxes: Monitoring Solr (JMX Edition) | AppNeta