You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2025/04/23 08:47:39 UTC

[I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

ChrisHegarty opened a new issue, #14546:
URL: https://github.com/apache/lucene/issues/14546

   ### Description
   
   ```
   TestForTooMuchCloning > test FAILED
       java.lang.AssertionError: too many calls to IndexInput.clone during TermRangeQuery: 7
           at __randomizedtesting.SeedInfo.seed([3B8D209D5C693E94:B3D91F47F295536C]:0)
           at org.junit.Assert.fail(Assert.java:89)
           at org.junit.Assert.assertTrue(Assert.java:42)
           at org.apache.lucene.index.TestForTooMuchCloning.test(TestForTooMuchCloning.java:88)
           at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
           at java.base/java.lang.reflect.Method.invoke(Method.java:580)
           at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
           at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
           at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
           at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
           at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
           at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
           at org.junit.rules.RunRules.evaluate(RunRules.java:20)
           at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
           at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
           at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
           at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
           at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
           at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
           at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
           at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
           at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
           at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
           at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
           at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
           at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
           at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
           at org.junit.rules.RunRules.evaluate(RunRules.java:20)
           at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
           at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
           at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
           at java.base/java.lang.Thread.run(Thread.java:1575)
   ```
   
   ### Gradle command to reproduce
   
   ./gradlew test --tests TestForTooMuchCloning.test -Dtests.seed=3B8D209D5C693E94 -Dtests.locale=kk-KZ -Dtests.timezone=America/Indiana/Marengo -Dtests.asserts=true -Dtests.file.encoding=UTF-8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2841363044

   Thanks for reporting @dweiss , i'll reopen.
   
   Error assert:
   
   https://github.com/apache/lucene/blob/2a5597e83b976cb11997285811d00565ff828ffc/lucene/core/src/test/org/apache/lucene/index/TestForTooMuchCloning.java#L65-L67
   
   This seems like an assert for the count of `IndexInput.clone` during merge, instead of query. I find the limitation 600 comes from https://github.com/apache/lucene/pull/14232, which looks like a "test passed after multiple reruns" magic number.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 closed issue #14546: TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7
URL: https://github.com/apache/lucene/issues/14546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2823593983

   git bisect tells me that this commit is the culprit #14380. /cc @gf2121 
   
   ```
   878e23f77390058b6787e0a0537de9087d1019b3 is the first bad commit
   commit 878e23f77390058b6787e0a0537de9087d1019b3
   Author: Guo Feng <52...@users.noreply.github.com>
   Date:   Mon Apr 14 23:19:04 2025 +0800
   
       A specialized Trie for Block Tree Index (#14380)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2824159527

   +1 to bump the clone limit to 7 so the test passes again -- clone inflation!!
   
   `clone` is likely super cheap for `MMapDirectory`?  Hmm it is the JDK version specific `MemorySegmentIndexInput` (with `SingleSegmentImpl` subclass to optimize the common case ) that is actually cloned, and it's making a new class instance, setting a few members, making a shallow copy of the actual mapped segments array, and (hmm) calling `.toString()` on the to-be-cloned `IndexInput` heh (this just returns the `resourceDescription` member).  So yeah it looks quite cheap.
   
   Also, this clone cost (fixed, one-time cost per segment) is amortized away as the query + segment matches more and more hits.  So crazy fast queries matching very few hits are a wee bit slower and still crazy fast, and slow queries matching many hits are not affected in any meaningful way ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2823836350

   Thanks @ChrisHegarty for git bisect and @rmuir for the context.
   
   (I allowed myself to edit your comment to point to the correct PR.)
   
   I knew the new trie will clone the indexInput twice instead of FST only once so I changed the limitation from 5 to 6 per segment, while i did not expect it will clone 7 times so much. I printed out all the clone traces:
   
   1. at org.apache.lucene.codecs.lucene103.blocktree.FieldReader.newReader(FieldReader.java:92)
   2. at org.apache.lucene.codecs.lucene103.blocktree.TrieReader.<init>(TrieReader.java:79)
   3. org.apache.lucene.codecs.lucene103.blocktree.IntersectTermsEnum.<init>(IntersectTermsEnum.java:85)
   4. org.apache.lucene.codecs.lucene103.blocktree.FieldReader.newReader(FieldReader.java:92)
   5. org.apache.lucene.codecs.lucene103.blocktree.TrieReader.<init>(TrieReader.java:79)
   6. at org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:83)
   7. org.apache.lucene.codecs.lucene103.Lucene103PostingsReader$BlockPostingsEnum.reset(Lucene103PostingsReader.java:517)
   
   So the `RangeQuery` intersect with `IntersectTermsEnum` (3 times), then `seeExact` with `SegmentTermsEnum` (3 times), and get postings (1 times). The TrieReader is constructed twice so we cloned twice more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2841306914

   The number of clones here seems... ahem, high.
   ```
     2> Apr 30, 2025 11:04:20 AM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
     2> WARNING: Vector bitsize and/or integer vectors enforcement; using default vectorization provider outside of testMode
      >     java.lang.AssertionError: too many calls to IndexInput.clone during merging: 612
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2823753882

   This test is old, i think. The background IIRC is that for some Directory impls (e.g. NIOFSDirectory or others that use "buffered input"), `clone()` may cause at minimum a 1KB read/buffer refill. For MMapDirectory the clone() may be cheaper and so excessive cloning may go unnoticed, yet cause a performance issue for some.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2824072414

   As the the added clones are cloning index file `tip` of term dictionary, which is usually opened with mmap. I raised a fix https://github.com/apache/lucene/pull/14547 to simply increase the limit of clones per segment from 6 to 7. Feel free to beat me if we think it is too many.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 closed issue #14546: TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7
URL: https://github.com/apache/lucene/issues/14546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2841305012

   I hit a failure on this again, after the patch:
   ```
   ./gradlew test -Ptests.seed=49BA5A550192C95E --tests TestForTooMuchCloning
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] TestForTooMuchCloning: too many calls to IndexInput.clone during TermRangeQuery: 7 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on issue #14546:
URL: https://github.com/apache/lucene/issues/14546#issuecomment-2842434087

   I raised:
   * https://github.com/apache/lucene/pull/14595 binding limitation to the number of segments get merged.
   * https://github.com/apache/lucene/pull/14596 a tiny simplification found when tracing clone stacks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org