[jira] [Updated] (SOLR-15859) Add handler to dump filter cache

["Shawn Heisey (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shawn Heisey updated SOLR-15859:
--------------------------------
    Status: Open  (was: Open)

[~ben.manes] This patch is my latest attempt to get a per-entry hitcounter by enabling a new "extraStats" parameter.

It's not working and I can't see what the problem is.  The patch includes some debug logging, each logging entry starts with BLIP so seeing the relevant logs is easy with grep.  If I can get it working, that logging will be removed before commit.

I seem to be running into Java's typical problems with  thread safety.  The core startup thread when extraStats is defined is setting extraStats to true and the threads where it is being accessed see it as false.  I tried an AtomicBoolean and a volatile boolean, and neither of those helped.

In the patch as you see it here, I wrapped everything having to do with extraStats or my hitcount Map in a "synchronized (this)" block and that isn't helping either.

In the logs I can see that "BLIP GET" and "BLIP GET INNER" are logged, but it never logs "BLIP GET INCREMENT" which is where updating the new stats would happen.  Also, in this latest patch the existing cache stats do not increase the overall hit counter, which is new.  At least one of the prior attempts did not interfere with the general stats, and I am not sure when those stopped working.

I'm betting that it's PEBCAK.

Thankfully all my attempts at least dump the cache keys, which gets [~petdance] the barebones info that allows troubleshooting.  I sent a new solr-core-9.0.0.jar that can be copied into a docker container.  Adding per-key hitcounts would make the idea even more useful.

If you (or anyone else) can see what I have done wrong, I would really appreciate knowing how to fix it.

This patch is against branch_9x but I think it will also apply on main, which I haven't yet tried.

> Add handler to dump filter cache
> --------------------------------
>
>                 Key: SOLR-15859
>                 URL: https://issues.apache.org/jira/browse/SOLR-15859
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Andy Lester
>            Assignee: Shawn Heisey
>            Priority: Major
>              Labels: FQ, cache, filtercache, metrics
>
> It would be very helpful to be able to inspect the contents of the filterCache.
> I'd like to be able to query something like
> {{/admin/caches?type=filter&nentries=1000&sort=numHits+DESC}}
> nentries would be allowed to be -1 to get everything.
> It would be nice to see these data items for each entry. I don't know which are available, but I'm thinking blue sky here:
>  * cache key, exactly as stored
>  * Timestamp when the entry was inserted
>  * Whether the insertion of the entry evicted another entry, and if so which one
>  * Timestamp of when this entry was last hit
>  * Number of hits on this entry forever
>  * Number of hits on this entry over some time period
>  * Number of documents matched by the filter
>  * Number of bytes of memory used by the filter
> These are the sorts of questions I'd like to be able answer:
>  * "I just did a query that I expect will have added a cache entry. Did it?"
>  * "Are my queries hitting existing cache entries?"
>  * "How big should I set my filterCache size? Should I limit it by number of entries or RAM usage?"
>  * "Which of my FQs are getting used the most? These are the ones I want in my firstSearcher queries." (I currently determine this by processing my old solr logs)
>  * "Which filters give me the most bang for the buck in terms of RAM usage?"
>  * "I have filter X and filter Y, but would it be beneficial if I made a filter X AND Y?"
>  * "Which FQs are used more at certain times of the day? (Assuming I take regular snapshots throughout the day)"
> I imagine a response might look like:
> {{{}}
> {{  "responseHeader": {}}
> {{    "status": 0,}}
> {{    "QTime": 961}}
> {{  },}}
> {{  "response": {}}
> {{    "numFound": 12104,}}
> {{    "filterCacheKeys": {}}
> {{      [}}
> {{        "language:eng": {}}
> {{          "inserted": "2021-12-04T07:34:16Z",}}
> {{          "lastHit": "2021-12-04T18:17:43Z",}}
> {{          "numHits": 15065,}}
> {{          "numHitsInPastHour": 2319,}}
> {{          "evictedKey": "agelevel:4 shippable:Y",}}
> {{          "numRecordsMatchedByFilter": 24328753,}}
> {{          "bytesUsed": 3041094}}
> {{        }}}
> {{      ],}}
> {{      [}}
> {{        "is_set:N": {}}
> {{          ...}}
> {{        }}}
> {{      ],}}
> {{      [}}
> {{        "language:spa": {}}
> {{          ...}}
> {{        }}}
> {{      ]}}
> {{    }}}
> {{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org