[jira] [Commented] (LUCENE-7410) Make cache keys and closed listeners less trappy

["Steve Rowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890516#comment-15890516 ] 

Steve Rowe commented on LUCENE-7410:
------------------------------------

My Jenkins found a reproducing seed for a {{TestReaderClosed.testReaderChaining()}} failure, and {{git bisect}} running the repro line says:

{noformat}
df6f83072303b4891a296b700a50c743284d3c30 is the first bad commit
commit df6f83072303b4891a296b700a50c743284d3c30
Author: Adrien Grand <jp...@gmail.com>
Date:   Tue Feb 28 14:21:30 2017 +0100

    LUCENE-7410: Make cache keys and close listeners less trappy.
{noformat}

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReaderClosed -Dtests.method=testReaderChaining -Dtests.seed=C4374342D2D99B8F -Dtests.slow=true -Dtests.locale=hi -Dtests.timezone=America/Dominica -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.04s J1 | TestReaderClosed.testReaderChaining <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Query failed, but not due to an AlreadyClosedException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C4374342D2D99B8F:530116CD6543D942]:0)
   [junit4]    > 	at org.apache.lucene.index.TestReaderClosed.testReaderChaining(TestReaderClosed.java:96)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, maxPointsInLeafNode=1885, maxMBSortInHeap=6.663525927605304, sim=RandomSimilarity(queryNorm=true): {}, locale=hi, timezone=America/Dominica
   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_77 (64-bit)/cpus=16,threads=1,free=82013152,total=289931264
{noformat}

> Make cache keys and closed listeners less trappy
> ------------------------------------------------
>
>                 Key: LUCENE-7410
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7410
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Adrien Grand
>             Fix For: master (7.0)
>
>         Attachments: LUCENE-7410.patch, LUCENE-7410.patch, LUCENE-7410.patch
>
>
> IndexReader currently exposes getCoreCacheKey(), getCombinedCoreAndDeletesKey(), addCoreClosedListener() and addReaderClosedListener(). They are typically used to manage resources whose lifetime needs to mimic the lifetime of segments/indexes, typically caches.
> I think this is trappy for various reasons:
> h3. Memory leaks
> When maintaining a cache, entries are added to the cache based on the cache key and then evicted using the cache key that is given back by the close listener, so it is very important that both keys are the same.
> But if a filter reader happens to delegate get*Key() and not add*ClosedListener() or vice-versa then there is potential for a memory leak since the closed listener will be called on a different key and entries will never be evicted from the cache.
> h3. Lifetime expectations
> The expectation of using the core cache key is that it will not change in case of deletions, but this is only true on SegmentReader and LeafReader impls that delegate to it. Other implementations such as composite readers or parallel leaf readers use the same key for "core" and "combined core and deletes".
> h3. Throw-away wrappers cause cache trashing
> An application might want to either expose more (with a ParrallelReader or MultiReader) or less information (by filtering fields/docs that can be seen) depending on the user who is logged in. In that case the application would typically maintain a DirectoryReader and then wrap it per request depending on the logged user and throw away the wrapper once the request is completed.
> The problem is that these wrappers have their own cache keys and the application may build something costly and put it in a cache to throw it away a couple milliseconds later. I would rather like for such readers to have a way to opt out from caching on order to avoid this performance trap.
> h3. Type safety
> The keys that are exposed are plain java.lang.Object instances, which requires caches to look like a {{Map<Object, ?>}} which makes it very easy to either try to get, put or remove on the wrong object since any object would be accepted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org