Possible bug with searchers and core swapping

[Shawn Heisey <ap...@elyograg.org>]

Today I noticed this on my dev server running Solr 5.2.1:

https://www.dropbox.com/s/bt81sv35acb7q2n/searcher-from-old-corename.png?dl=0

The name of this core is spark0live, but before the last index rebuild,
it was named spark0build.  My full rebuild process indexes data into
build cores, then when the rebuild is done, it swaps the build cores
with the live cores.

This core is holding on to the searcher object from before the core
swap.  The last rebuild I did was days ago, so it was getting pretty old.

A quick test on a 5.3.1 download (not a full test with my config/schema)
did not reveal any problems.

When I did some manual testing with core swaps on the 5.2.1 dev server,
I could not reproduce the problem.

On the index with the problem, I am using one third-party plugin with a
schema component and an update processor, and an update processor that I
wrote myself.  Is it possible that these might be causing this to happen
if objects are not being closed properly?

I've since restarted the server, so I will need to initiate a full
rebuild to see if the problem comes back.

Thanks,
Shawn

Re: Possible bug with searchers and core swapping

[Shawn Heisey <ap...@elyograg.org>]

On 10/23/2015 5:27 PM, Shawn Heisey wrote:
> Today I noticed this on my dev server running Solr 5.2.1:
>
> https://www.dropbox.com/s/bt81sv35acb7q2n/searcher-from-old-corename.png?dl=0

<snip>

> On the index with the problem, I am using one third-party plugin with a
> schema component and an update processor, and an update processor that I
> wrote myself.  Is it possible that these might be causing this to happen
> if objects are not being closed properly?

If I reload the core, then the extra searcher disappears.  The problem
persists after doing soft commits on the core, so a commit doesn't
help.  If I manually swap cores rather than doing it as part of a full
index rebuild, this problem does not occur, which is very confusing.

The problem happened on an index that has the custom components defined,
but nothing in the index data actually triggers their use.  I can try
removing those components completely and rebuilding that index again.

When it was running on Java 8, I grabbed a heap dump, but there's a bug
in Eclipse MAT that prevents opening heap dumps created by Java 8, so I
restarted with Java 7 and found the same problem after rebuilding the
index.  Now I have a heap dump that I can open in the Eclipse
MemoryAnalyzer.  If anyone knows how to find these Searcher objects in
the MAT, I would appreciate knowing.  The object I am currently looking
for says "Searcher@3cd076ad[ai-0build] main" in the admin UI.  There
does not appear to be an object in the source code called "Searcher" so
the Plugins/Stats page in the UI seems to be not reporting true object
names.  I think that's something that needs to be fixed.

Thanks,
Shawn