What do you usually look for in Solr logs?

[Radu Gheorghe <ra...@sematext.com>]

Hello Solr users,

I've recently added a Solr logs integration
<https://sematext.com/docs/integration/solr-logs/> to our logging SaaS
<https://sematext.com/docs/logs/> and I wanted to ask what would be useful
that I may have missed.

First, there are some regexes to parse Solr Logs here:
https://github.com/sematext/logagent-js/blob/master/patterns.yml#L140-L379
BTW, Logagent is open-source, so you can use it not only with our SaaS, but
with your own Elasticsearch (at least until there's an output plugin for
Solr :D) or you can simply reuse the regexes.

I've tested parsing on a number of Solr versions and they should work on
7.x and later. But what about the extracted fields? Do you see something
interesting that's not already there?

On the frontend side, what do you find useful? Because we have some default
dashboards and we can always add/update them. Let me quickly tell you
what's already included.

The first one you'd see, called Overview, is quite self-explanatory:


The one I use most is called Queries. All widgets have a filter by (+path:*
-path:admin -path:update) and most of them are built around QTime and Hits
metrics:


I won't spam you with more screenshots, but there are other dashboards, too:
- *Errors*: quite similar to Overview, but with a (severity:error) filter
- *Zookeeper*: I wanted to catch ZK communication here, so the filter is
(thread:zk* OR class:zk* OR class.raw:o.a.z.*) and the "usual" breakdown
widgets e.g. by host or severity
- *Overseer*: similar to Zookeeper, but filtering on (thread:overseer* OR
class:ShardLeaderElection* OR class:SyncStrategy)
- *Audit*: this is for Solr Audit Logging
<https://lucene.apache.org/solr/guide/8_6/audit-logging.html>, filtering on
(class.raw:"o.a.s.s.SolrLogAuditLoggerPlugin") Besides the usual
breakdowns, there are audit-specific widgets, like request types over time
- *Start & stop*: I'm trying to catch startup- and shutdown-specific logs
here. The best filters I found are (thread:main) for startup and
(thread:ShutdownMonitor OR thread:closeThreadPool* OR
thread:coreCloseExecutor*) for shutdown. Do you see better criteria?

Last but not least, what do you usually look for in Solr logs? Anything
that we don't cover in the above? Any feedback will be very much
appreciated!

Thanks and best regards,
Radu
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/