You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "shubhamsrkdev (via GitHub)" <gi...@apache.org> on 2026/01/16 22:27:36 UTC

[PR] Added shared prefetch counter [lucene]

shubhamsrkdev opened a new pull request, #15585:
URL: https://github.com/apache/lucene/pull/15585

   ### Problem
   Across clone and slices of `MemorySegmentIndexInput` the consecutivePrefetchHitCount was getting reset leading to repeated calls.
   
   ### Solution
   Added a shared prefetch counter, through reference - this should maintain count through clones and slices.
   
   Fixes #15571 
   
   The benchmarks are looking good :
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                             IntSet      814.31      (4.5%)      794.78      (6.2%)   -2.4% ( -12% -    8%) 0.159
                         TermDTSort      248.72      (5.8%)      245.11      (7.0%)   -1.5% ( -13% -   12%) 0.476
                    LowSloppyPhrase       38.85      (3.9%)       38.37      (4.7%)   -1.3% (  -9% -    7%) 0.360
               BrowseDateSSDVFacets        0.90      (8.4%)        0.89      (9.0%)   -0.8% ( -16% -   18%) 0.767
                    MedSloppyPhrase       45.58      (3.0%)       45.22      (3.4%)   -0.8% (  -6% -    5%) 0.445
                   HighSloppyPhrase       12.27      (6.0%)       12.20      (5.9%)   -0.6% ( -11% -   12%) 0.749
                             IntNRQ      295.84      (1.7%)      294.19      (1.7%)   -0.6% (  -3% -    2%) 0.302
                            Respell       45.97      (2.5%)       45.77      (1.9%)   -0.4% (  -4% -    4%) 0.539
              BrowseMonthTaxoFacets        2.78      (2.0%)        2.77      (0.9%)   -0.4% (  -3% -    2%) 0.455
                       HighSpanNear       27.73      (1.5%)       27.63      (1.4%)   -0.3% (  -3% -    2%) 0.471
                             Fuzzy2       68.17      (2.6%)       67.95      (2.1%)   -0.3% (  -4% -    4%) 0.668
               HighIntervalsOrdered       32.46      (8.2%)       32.42      (9.3%)   -0.1% ( -16% -   18%) 0.965
                              range     3603.08      (4.7%)     3600.78      (4.9%)   -0.1% (  -9% -    9%) 0.966
        BrowseRandomLabelSSDVFacets        3.22      (8.7%)        3.22      (8.1%)   -0.0% ( -15% -   18%) 0.996
             OrHighMedDayTaxoFacets        4.69      (1.9%)        4.69      (1.9%)    0.0% (  -3% -    3%) 0.975
              HighTermDayOfYearSort      365.33      (1.4%)      365.41      (1.2%)    0.0% (  -2% -    2%) 0.959
                             Fuzzy1       70.44      (2.4%)       70.48      (1.8%)    0.1% (  -4% -    4%) 0.940
                           PKLookup      197.00      (1.7%)      197.17      (2.2%)    0.1% (  -3% -    4%) 0.893
        BrowseRandomLabelTaxoFacets        2.39      (1.9%)        2.39      (2.0%)    0.2% (  -3% -    4%) 0.753
               HighTermTitleBDVSort       63.91      (1.2%)       64.09      (1.0%)    0.3% (  -1% -    2%) 0.427
              BrowseMonthSSDVFacets        4.52      (6.4%)        4.53      (7.2%)    0.3% ( -12% -   14%) 0.883
          BrowseDayOfYearSSDVFacets        4.52      (8.1%)        4.53      (7.7%)    0.4% ( -14% -   17%) 0.888
                MedIntervalsOrdered       61.93      (4.5%)       62.17      (5.3%)    0.4% (  -9% -   10%) 0.811
                        MedSpanNear      149.61      (1.3%)      150.36      (1.0%)    0.5% (  -1% -    2%) 0.167
                        LowSpanNear      102.70      (1.2%)      103.25      (0.8%)    0.5% (  -1% -    2%) 0.099
            AndHighMedDayTaxoFacets       71.17      (0.8%)       71.56      (0.9%)    0.5% (  -1% -    2%) 0.053
                LowIntervalsOrdered      273.02      (3.2%)      274.68      (4.2%)    0.6% (  -6% -    8%) 0.608
                          LowPhrase       54.15      (2.4%)       54.48      (1.7%)    0.6% (  -3% -    4%) 0.349
           AndHighHighDayTaxoFacets       35.01      (0.9%)       35.28      (0.9%)    0.8% (  -1% -    2%) 0.006
               MedTermDayTaxoFacets       35.19      (1.2%)       35.51      (0.8%)    0.9% (  -1% -    2%) 0.003
          BrowseDayOfYearTaxoFacets        3.12      (4.1%)        3.15      (4.6%)    1.0% (  -7% -   10%) 0.480
               BrowseDateTaxoFacets        3.10      (4.1%)        3.13      (4.7%)    1.0% (  -7% -   10%) 0.456
                         HighPhrase       87.16      (2.9%)       88.09      (1.9%)    1.1% (  -3% -    6%) 0.176
                        AndHighHigh      261.20      (2.9%)      266.49      (2.8%)    2.0% (  -3% -    7%) 0.025
                          MedPhrase      256.22      (1.8%)      261.49      (1.5%)    2.1% (  -1% -    5%) 0.000
                  HighTermMonthSort     1355.00      (2.3%)     1386.97      (3.3%)    2.4% (  -3% -    8%) 0.009
                       OrNotHighMed      585.83      (3.3%)      603.68      (3.9%)    3.0% (  -4% -   10%) 0.008
                         OrHighHigh      309.70      (5.2%)      319.44      (3.8%)    3.1% (  -5% -   12%) 0.029
                           Wildcard      815.29      (3.2%)      845.21      (2.9%)    3.7% (  -2% -   10%) 0.000
                          OrHighMed      626.92      (3.0%)      654.05      (2.1%)    4.3% (   0% -    9%) 0.000
                  HighTermTitleSort      149.65      (2.8%)      156.38      (4.8%)    4.5% (  -3% -   12%) 0.000
                            MedTerm     1168.83      (4.1%)     1222.83      (5.0%)    4.6% (  -4% -   14%) 0.001
                      OrHighNotHigh      728.82      (3.9%)      762.67      (4.1%)    4.6% (  -3% -   13%) 0.000
                      OrNotHighHigh      867.08      (2.8%)      907.92      (3.7%)    4.7% (  -1% -   11%) 0.000
                           HighTerm      734.14      (5.8%)      769.02      (5.5%)    4.8% (  -6% -   16%) 0.008
                       OrHighNotMed      858.31      (3.1%)      901.62      (3.4%)    5.0% (  -1% -   11%) 0.000
                         AndHighMed      573.81      (2.6%)      603.39      (1.6%)    5.2% (   0% -    9%) 0.000
                       OrNotHighLow     1102.50      (2.7%)     1165.73      (2.8%)    5.7% (   0% -   11%) 0.000
                            LowTerm     1691.06      (3.5%)     1822.95      (3.7%)    7.8% (   0% -   15%) 0.000
                          OrHighLow     1074.51      (2.6%)     1159.81      (2.5%)    7.9% (   2% -   13%) 0.000
                       OrHighNotLow     1076.27      (4.0%)     1167.56      (5.4%)    8.5% (   0% -   18%) 0.000
                         AndHighLow     1250.35      (2.1%)     1374.26      (3.3%)    9.9% (   4% -   15%) 0.000
                            Prefix3      742.64     (16.9%)      861.01     (20.1%)   15.9% ( -18% -   63%) 0.007
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805845991

   > I'll try to backport to 10.x soon ... something's wrong w/ my 10.x dev area .
   
   D'oh!  I was asking gradle to:
   
   ```
     ./gradlew clean check
   ```
   
   (both at once) and that (weirdly) leads to the confusing deadlock.  Why can't it just say "I'm sorry I cannot do both at once like that".  Whereas if I separate them:
   
   ```
     ./gradlew clean
     ./gradlew check
   ```
   
   'Tsall good!
   
   I think this weird gradle bug only happens in `10.x` not `main` branch -- @rmuir thinks @dweiss fixed it already in `main`.  But @rmuir suggested switching to the nuclear `git clean -xfd` option instead of trusting gradle for cleaning -- I'll try to do that from now on!  Gotta erase my muscle memory...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3810828129

   Hi,
   
   > Thanks @shubhamsrkdev -- I think this is ready. I'll merge soon. Let's watch builds closely... I worry a bit about the GC implications of this change, for Lucene usage that has a great many clones (many queries in flight, GC being lazy since there's lots of heap headroom say) and all of those clones pointing to this one `AtomicInteger`... likely it's fine but we've had GC issues in the past when `MMapDirectory` tried to carefully track clones (before Panama gave us better mmap APIs -- no more scary `Unsafe` for unmapping).
   
   this is not an issue here. The clones only share the same AtomicInteger. Basically all clones also refer to the same Arena (which is grouped, too so has additional stuff). Basically this is all fine, the problems with tracking clones were a different category of issues because we needed a pointer from the parent input to its childs and this goes worse. Here we just refer to some same object, which itsself has no reference to the clones. Once refcounting is zero the atomic integer gets free for GC. The problem with the master refering to childs was that as ong as the master input refers to its childs, those cannot be GCed. As we never call close on clones there is no way to figure out for the master that it is gone. The hard relationship master -> clone was the killer.
   
   Uwe


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704668163


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
  * chunkSizePower</code>).
  */
 abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+  /** Shared counter for prefetch hit tracking. */
+  static final class SharedPrefetchCounter {
+    private int count;

Review Comment:
   Should we make count `volatile` to ensure visibility across threads?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3790523874

   > Which corpus was the luceneutil benchy results run on? Crazy how fast Lucene queries have become!
   
   This was on `wikimediumall` . I will mention this in the description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721591904


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
       boolean confined,
       Function<IOContext, ReadAdvice> toReadAdvice) {
     assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+    AtomicInteger sharedPrefetchCounter = new AtomicInteger();

Review Comment:
   >Is this creating one AtomicInteger per actually originally opened files? And then each clone shares that same original AtomicInteger?
   
   Yes!
   
   >Hmm what happens when we overflow the int? What does incrementAndGet do then? Silently overflow/wraparound to negative?
   
   It does wraparound after `Integer.MAX_VALUE` to Integer.MIN_VALUE and subsequent negative values. But it would just cause an extra prefetch because of `isZeroOrPowerOfTwo` ?`isZeroOrPowerOfTwo(Integer.MIN_VALUE)` would give true but subsequent increments would just give false as expected. For example:
   
   ```
   import java.util.concurrent.atomic.AtomicInteger;
   
    class Main {
       public static void main(String[] args) {
           AtomicInteger counter = new AtomicInteger(Integer.MAX_VALUE);
   
           while(counter.incrementAndGet() < 0)
           {
               System.out.println(isZeroOrPowerOfTwo(counter.get()));
               counter = new AtomicInteger(counter.incrementAndGet());
           }
       }
       
        
       public static boolean isZeroOrPowerOfTwo(int x) {
       return (x & (x - 1)) == 0;
     }
   }
   ```
   
   This would give:
   
   ```
   true
   false
   false
   false
   false
   false
   false
   false
   false
   false
   false
   .
   .
   .
   .
   .
   .
   .
   .
   .
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3819258882

   > (both at once) and that (weirdly) leads to the confusing deadlock. Why can't it just say "I'm sorry I cannot do both at once like that".
   
   It's complicated - there are included builds (composite builds) and they're treated differently... chicken and egg. 
   
   > But @rmuir suggested switching to the nuclear `git clean -xfd` option instead of trusting gradle for cleaning 
   
   It's also much faster... My choice too, really.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2718698084


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -316,7 +335,7 @@ public void prefetch(long offset, long length) throws IOException {
 
     ensureOpen();
 
-    if (BitUtil.isZeroOrPowerOfTwo(consecutivePrefetchHitCount++) == false) {
+    if (BitUtil.isZeroOrPowerOfTwo(sharedPrefetchCounter.incrementAndGet()) == false) {

Review Comment:
   Does it matter that before it was get-and-increment and now it's increment-and-get?  E.g. we will now start with 0 and get `true` the first time, but previously false, so won't we fail to check the first time we come through with this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721595114


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -316,7 +335,7 @@ public void prefetch(long offset, long length) throws IOException {
 
     ensureOpen();
 
-    if (BitUtil.isZeroOrPowerOfTwo(consecutivePrefetchHitCount++) == false) {
+    if (BitUtil.isZeroOrPowerOfTwo(sharedPrefetchCounter.incrementAndGet()) == false) {

Review Comment:
   Good catch! I can change this to `getAndIncrement()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3786754721

   Which corpus was the luceneutil benchy results run on?  Crazy how fast Lucene queries have become!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805698102

   I'll try to backport to 10.x soon ... something's wrong w/ my 10.x dev area ... I think this is gradle solving the halting problem (confessing it is stuck / will never halt) for Lucene's gradle config, and reporting it in a very unintuitive manner :) ... I'll try a fresh clone (I did kill its daemons already):
   
   ```
   > Task :missing-doclet:jar
   Unable to make progress running work. The following items are queued for execution but none of them can be started:
     - Build ':':
         - Waiting for nodes:
             - :lucene:spatial3d:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:spatial3d:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:spatial3d:test (SHOULD_RUN)] )
             - :lucene:analysis.tests:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:analysis.tests:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:analysis.tests:test (SHOULD_RUN)] )
             - :lucene:memory:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:memory:ecjLint (SHOULD_RUN), :lucene:memory:forbiddenApis (SHOULD_RUN), :lucene:memory:licenses (SHOULD_RUN), :lucene:memory:spotlessCheck (SHOULD_RUN), :lucene:memory:test (SHOULD_RUN), :lucene:memory:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:m\
   emory:licenses (SHOULD_RUN), :lucene:memory:ecjLint (SHOULD_RUN), :lucene:memory:forbiddenApis (SHOULD_RUN), :lucene:memory:validateSourcePatterns (SHOULD_RUN), :lucene:memory:test (SHOULD_RUN), :lucene:memory:spotlessCheck (SHOULD_RUN)], has-failed-dependency=false )
             - :lucene:backward-codecs:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:backward-codecs:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:backward-codecs:test (SHOULD_RUN)] )
             - :lucene:analysis:icu:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:analysis:icu:ecjLint (SHOULD_RUN), :lucene:analysis:icu:forbiddenApis (SHOULD_RUN), :lucene:analysis:icu:genRbbiChecksumCheck (EXECUTED), :lucene:analysis:icu:genUtr30DataFilesChecksumCheck (EXECUTED), :lucene:analysis:icu:licenses (SHOULD_RUN), :lucene:an\
   alysis:icu:spotlessCheck (SHOULD_RUN), :lucene:analysis:icu:test (SHOULD_RUN), :lucene:analysis:icu:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:analysis:icu:spotlessCheck (SHOULD_RUN), :lucene:analysis:icu:validateSourcePatterns (SHOULD_RUN), :lucene:analysis:icu:forbiddenApis (SHOULD_RUN), :lucene:analysis:icu:test (SHOULD_RUN), :lucene:analysis:icu:ecjLint (SHOULD\
   _RUN), :lucene:analysis:icu:licenses (SHOULD_RUN)], has-failed-dependency=false )
             - :lucene:queryparser:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:queryparser:ecjLint (SHOULD_RUN), :lucene:queryparser:forbiddenApis (SHOULD_RUN), :lucene:queryparser:javaccParserClassicChecksumCheck (EXECUTED), :lucene:queryparser:javaccParserFlexibleChecksumCheck (EXECUTED), :lucene:queryparser:javaccParserSurroundChec\
   ksumCheck (EXECUTED), :lucene:queryparser:licenses (SHOULD_RUN), :lucene:queryparser:spotlessCheck (SHOULD_RUN), :lucene:queryparser:test (SHOULD_RUN), :lucene:queryparser:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:queryparser:forbiddenApis (SHOULD_RUN), :lucene:queryparser:validateSourcePatterns (SHOULD_RUN), :lucene:queryparser:ecjLint (SHOULD_RUN), :lucene:query\
   parser:licenses (SHOULD_RUN), :lucene:queryparser:spotlessCheck (SHOULD_RUN), :lucene:queryparser:test (SHOULD_RUN)], has-failed-dependency=false )
             - :lucene:join:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:join:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:join:test (SHOULD_RUN)] )
             - :lucene:analysis:morfologik:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:analysis:morfologik:ecjLint (SHOULD_RUN), :lucene:analysis:morfologik:forbiddenApis (SHOULD_RUN), :lucene:analysis:morfologik:licenses (SHOULD_RUN), :lucene:analysis:morfologik:spotlessCheck (SHOULD_RUN), :lucene:analysis:morfologik:test (SHOULD_RUN\
   ), :lucene:analysis:morfologik:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:analysis:morfologik:spotlessCheck (SHOULD_RUN), :lucene:analysis:morfologik:validateSourcePatterns (SHOULD_RUN), :lucene:analysis:morfologik:licenses (SHOULD_RUN), :lucene:analysis:morfologik:test (SHOULD_RUN), :lucene:analysis:morfologik:ecjLint (SHOULD_RUN), :lucene:analysis:morfologik:forb\
   iddenApis (SHOULD_RUN)], has-failed-dependency=false )
             - :lucene:highlighter:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:highlighter:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:highlighter:test (SHOULD_RUN)] )
             - :lucene:analysis:morfologik:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:analysis:morfologik:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:analysis:morfologik:test (SHOULD_RUN)] )
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721591904


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
       boolean confined,
       Function<IOContext, ReadAdvice> toReadAdvice) {
     assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+    AtomicInteger sharedPrefetchCounter = new AtomicInteger();

Review Comment:
   >Is this creating one AtomicInteger per actually originally opened files? And then each clone shares that same original AtomicInteger?
   
   Yes!
   
   >Hmm what happens when we overflow the int? What does incrementAndGet do then? Silently overflow/wraparound to negative?
   
   It does wraparound after `Integer.MAX_VALUE` to `Integer.MIN_VALUE` and subsequent negative values. But it would just cause an extra prefetch because of `isZeroOrPowerOfTwo` ?`isZeroOrPowerOfTwo(Integer.MIN_VALUE)` would give true but subsequent increments would just give false as expected. For example:
   
   ```
   import java.util.concurrent.atomic.AtomicInteger;
   
    class Main {
       public static void main(String[] args) {
           AtomicInteger counter = new AtomicInteger(Integer.MAX_VALUE);
   
           while(counter.incrementAndGet() < 0)
           {
               System.out.println(isZeroOrPowerOfTwo(counter.get()));
           }
       }
       
        
       public static boolean isZeroOrPowerOfTwo(int x) {
       return (x & (x - 1)) == 0;
     }
   }
   ```
   
   This would give:
   
   ```
   true
   false
   false
   false
   false
   false
   false
   false
   false
   false
   false
   .
   .
   .
   .
   .
   .
   .
   .
   .
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3810843016

   Basically everything is shared between master and clone (including the memory segments, and their arena - implicit) - the clones and master only differ in their position and slice coordinates.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2705127972


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
  * chunkSizePower</code>).
  */
 abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+  /** Shared counter for prefetch hit tracking. */
+  static final class SharedPrefetchCounter {
+    private int count;

Review Comment:
   I suggested volatile because I don't think the counter needs to be perfectly precise, and using thread-safe types like AtomicInteger or LongAdder might increase thread contention. That said, perhaps we could test all three options (current implementation, volatile, AtomicInteger) and choose the fastest one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704817314


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
  * chunkSizePower</code>).
  */
 abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+  /** Shared counter for prefetch hit tracking. */
+  static final class SharedPrefetchCounter {
+    private int count;

Review Comment:
   or LongAdder  which is designed exactly for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704974447


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
  * chunkSizePower</code>).
  */
 abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+  /** Shared counter for prefetch hit tracking. */
+  static final class SharedPrefetchCounter {
+    private int count;

Review Comment:
   Maybe I can use `AtomicInteger` ? Should cover thread safety... `LongAdder` doesn't have an increment and get, maybe it is better for less read intensive scenarios?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2732151120


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
       boolean confined,
       Function<IOContext, ReadAdvice> toReadAdvice) {
     assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+    AtomicInteger sharedPrefetchCounter = new AtomicInteger();

Review Comment:
   So net/net I think the behavior is reasonable -- we check with 1/2 the frequency, up until `1 << 31 = 2147483648` at which point we stop halving the frequency of checks and do it every `2147483648` calls.  That's great -- no exception, still checking (rarely!).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805633205

   > The benchmarks are looking good :
   
   If I understand correctly, Lucene `main` now defaults to no IOContext-based `madvise`?  And luceneutil just goes with Lucene's defaults here?  Which makes these benchmark results very surprising!  Oh, wait!, that is a different `madvise`.
   
   This change is about Lucene's dynamic prefetch -- `madvise(WILL_NEED)` for specific byte ranges of the file.  (Lucene also has static prefetch -- up front `madvise(WILL_NEED)` on the entire file when it's opened, but we are not changing that here).  So this change fixes the frequency of clones calling `.isLoaded()` on the entire file.  In the hot case, this is disastrous (slowest call for `.isLoaded()`).
   
   Still, for `HighTerm`, which is a high frequency term, we create the clones (once per segment) and then use them for a long-ish time.  I would've thought that the high `.isLoaded()` cost would've been amortized and we wouldn't see such gains from `HighTerm`.  Confusing...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2705209143


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
  * chunkSizePower</code>).
  */
 abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+  /** Shared counter for prefetch hit tracking. */
+  static final class SharedPrefetchCounter {
+    private int count;

Review Comment:
   Yes, I did that - looks like they are almost the same `AtomicInteger` being somewhat better?
   
   - `volatile`
   
   ```
                              TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
          BrowseDayOfYearTaxoFacets        3.28      (9.4%)        3.20      (4.8%)   -2.6% ( -15% -   12%) 0.280
               BrowseDateTaxoFacets        3.24      (7.9%)        3.18      (4.9%)   -2.0% ( -13% -   11%) 0.331
                         TermDTSort      261.53      (3.2%)      259.46      (3.5%)   -0.8% (  -7% -    6%) 0.457
                              range     3676.85      (4.3%)     3660.49      (5.1%)   -0.4% (  -9% -    9%) 0.764
        BrowseRandomLabelTaxoFacets        2.40      (4.3%)        2.39      (2.0%)   -0.4% (  -6% -    6%) 0.691
                MedIntervalsOrdered      147.92      (6.6%)      147.43      (6.4%)   -0.3% ( -12% -   13%) 0.872
               HighIntervalsOrdered       17.40      (8.7%)       17.35      (9.0%)   -0.3% ( -16% -   19%) 0.914
                      OrHighNotHigh      371.74      (5.7%)      370.95      (6.3%)   -0.2% ( -11% -   12%) 0.911
                      OrNotHighHigh      266.13      (6.5%)      265.65      (6.8%)   -0.2% ( -12% -   13%) 0.931
                             IntNRQ      466.76      (1.9%)      466.10      (1.9%)   -0.1% (  -3% -    3%) 0.812
                       HighSpanNear       34.12      (3.3%)       34.11      (4.6%)   -0.0% (  -7% -    8%) 0.991
                LowIntervalsOrdered      116.50      (2.5%)      116.49      (2.5%)   -0.0% (  -4% -    5%) 0.991
             OrHighMedDayTaxoFacets        7.60      (1.9%)        7.60      (1.9%)    0.1% (  -3% -    3%) 0.860
               HighTermTitleBDVSort       31.69      (2.3%)       31.80      (2.2%)    0.3% (  -4% -    4%) 0.625
              BrowseMonthTaxoFacets        2.75      (1.7%)        2.76      (0.3%)    0.4% (  -1% -    2%) 0.356
           AndHighHighDayTaxoFacets       29.66      (1.4%)       29.77      (1.2%)    0.4% (  -2% -    2%) 0.387
                   HighSloppyPhrase       35.65      (2.0%)       35.78      (2.6%)    0.4% (  -4% -    5%) 0.625
                    LowSloppyPhrase      117.98      (2.5%)      118.53      (3.2%)    0.5% (  -5% -    6%) 0.611
                           PKLookup      198.36      (2.2%)      199.34      (1.7%)    0.5% (  -3% -    4%) 0.424
                    MedSloppyPhrase       52.11      (2.3%)       52.37      (2.9%)    0.5% (  -4% -    5%) 0.542
              BrowseMonthSSDVFacets        4.65     (14.6%)        4.67     (15.4%)    0.5% ( -25% -   35%) 0.915
              HighTermDayOfYearSort      354.00      (1.7%)      356.01      (1.4%)    0.6% (  -2% -    3%) 0.251
                        MedSpanNear      118.13      (2.8%)      118.85      (3.4%)    0.6% (  -5% -    7%) 0.540
               MedTermDayTaxoFacets       17.07      (2.1%)       17.18      (1.7%)    0.6% (  -3% -    4%) 0.297
                            Respell       40.13      (1.8%)       40.38      (1.8%)    0.6% (  -2% -    4%) 0.266
            AndHighMedDayTaxoFacets       29.22      (1.1%)       29.43      (1.2%)    0.7% (  -1% -    3%) 0.051
                         HighPhrase       54.55      (1.4%)       54.95      (1.5%)    0.7% (  -2% -    3%) 0.107
                            Prefix3      149.74      (5.4%)      151.03      (3.4%)    0.9% (  -7% -   10%) 0.549
                          MedPhrase      155.10      (1.2%)      156.71      (1.4%)    1.0% (  -1% -    3%) 0.013
                        LowSpanNear      168.63      (3.2%)      170.49      (3.9%)    1.1% (  -5% -    8%) 0.324
                             Fuzzy1       80.40      (3.3%)       81.62      (2.9%)    1.5% (  -4% -    7%) 0.123
                             Fuzzy2       78.61      (3.9%)       79.84      (3.2%)    1.6% (  -5% -    8%) 0.163
                       OrNotHighMed      542.60      (4.8%)      551.16      (4.2%)    1.6% (  -7% -   11%) 0.266
                             IntSet      794.07      (3.9%)      806.99      (3.1%)    1.6% (  -5% -    8%) 0.142
                  HighTermTitleSort      129.70      (2.8%)      131.85      (2.8%)    1.7% (  -3% -    7%) 0.059
                        AndHighHigh      345.95      (3.0%)      352.27      (5.9%)    1.8% (  -6% -   10%) 0.214
          BrowseDayOfYearSSDVFacets        4.63     (14.3%)        4.71     (16.4%)    1.9% ( -25% -   38%) 0.694
                         OrHighHigh      407.65      (3.1%)      416.03      (5.8%)    2.1% (  -6% -   11%) 0.161
                       OrHighNotMed      789.81      (4.1%)      806.23      (6.1%)    2.1% (  -7% -   12%) 0.206
        BrowseRandomLabelSSDVFacets        3.09      (4.5%)        3.17      (9.7%)    2.5% ( -11% -   17%) 0.305
               BrowseDateSSDVFacets        0.91      (7.8%)        0.93      (5.8%)    2.5% ( -10% -   17%) 0.244
                          OrHighMed      579.17      (2.2%)      598.98      (4.3%)    3.4% (  -3% -   10%) 0.002
                         AndHighMed      529.81      (1.6%)      548.32      (3.6%)    3.5% (  -1% -    8%) 0.000
                  HighTermMonthSort     1362.64      (2.9%)     1420.03      (2.8%)    4.2% (  -1% -   10%) 0.000
                           Wildcard      788.01      (3.1%)      821.42      (3.4%)    4.2% (  -2% -   11%) 0.000
                          LowPhrase      572.79      (1.4%)      599.26      (1.3%)    4.6% (   1% -    7%) 0.000
                           HighTerm     1037.42      (2.5%)     1085.74      (6.2%)    4.7% (  -3% -   13%) 0.002
                            MedTerm     1140.70      (3.7%)     1198.80      (5.0%)    5.1% (  -3% -   14%) 0.000
                       OrHighNotLow     1093.24      (3.5%)     1157.63      (4.9%)    5.9% (  -2% -   14%) 0.000
                            LowTerm     1595.88      (2.6%)     1714.75      (7.5%)    7.4% (  -2% -   18%) 0.000
                          OrHighLow     1027.76      (2.5%)     1107.88      (2.1%)    7.8% (   3% -   12%) 0.000
                       OrNotHighLow     1456.36      (2.5%)     1575.86      (2.5%)    8.2% (   3% -   13%) 0.000
                         AndHighLow     1279.96      (3.2%)     1418.90      (3.5%)   10.9% (   3% -   18%) 0.000
   ```
   
   - `AtomicInteger`
   
   
   ```
   TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                             IntSet      824.23      (5.1%)      812.38      (5.1%)   -1.4% ( -11% -    9%) 0.371
                             Fuzzy2       79.65      (3.1%)       78.82      (3.7%)   -1.0% (  -7% -    5%) 0.335
                             Fuzzy1       71.11      (2.6%)       70.48      (2.9%)   -0.9% (  -6% -    4%) 0.312
               MedTermDayTaxoFacets       40.99      (1.1%)       40.74      (1.1%)   -0.6% (  -2% -    1%) 0.073
                         TermDTSort      255.15      (3.2%)      253.86      (2.6%)   -0.5% (  -6% -    5%) 0.588
                            Respell       33.25      (1.7%)       33.09      (2.3%)   -0.5% (  -4% -    3%) 0.437
           AndHighHighDayTaxoFacets       16.63      (1.4%)       16.57      (1.7%)   -0.4% (  -3% -    2%) 0.429
                             IntNRQ      438.66      (1.9%)      437.29      (1.4%)   -0.3% (  -3% -    3%) 0.557
                    LowSloppyPhrase       33.33      (4.3%)       33.32      (4.9%)   -0.0% (  -8% -    9%) 0.980
               HighTermTitleBDVSort       74.23      (1.5%)       74.24      (1.7%)    0.0% (  -3% -    3%) 0.973
              BrowseMonthTaxoFacets        2.77      (1.5%)        2.77      (1.5%)    0.0% (  -2% -    3%) 0.961
               BrowseDateSSDVFacets        0.87     (10.0%)        0.87      (9.3%)    0.1% ( -17% -   21%) 0.985
                           PKLookup      198.31      (1.9%)      198.66      (2.3%)    0.2% (  -3% -    4%) 0.789
              HighTermDayOfYearSort      293.62      (1.8%)      294.17      (2.2%)    0.2% (  -3% -    4%) 0.767
               BrowseDateTaxoFacets        3.15      (4.8%)        3.15      (4.7%)    0.2% (  -8% -   10%) 0.872
          BrowseDayOfYearTaxoFacets        3.17      (4.7%)        3.18      (4.6%)    0.3% (  -8% -   10%) 0.862
             OrHighMedDayTaxoFacets       11.82      (2.4%)       11.85      (1.9%)    0.3% (  -3% -    4%) 0.702
                   HighSloppyPhrase       25.82      (3.4%)       25.91      (4.3%)    0.3% (  -7% -    8%) 0.788
        BrowseRandomLabelTaxoFacets        2.38      (1.7%)        2.39      (1.9%)    0.3% (  -3% -    3%) 0.540
        BrowseRandomLabelSSDVFacets        3.22     (11.0%)        3.24     (10.4%)    0.7% ( -18% -   24%) 0.842
                       OrNotHighMed      402.90      (6.4%)      405.86      (6.8%)    0.7% ( -11% -   14%) 0.726
                       HighSpanNear       41.21      (2.7%)       41.54      (2.6%)    0.8% (  -4% -    6%) 0.345
                    MedSloppyPhrase      120.35      (2.8%)      121.37      (3.4%)    0.8% (  -5% -    7%) 0.400
            AndHighMedDayTaxoFacets      102.67      (0.7%)      103.56      (0.7%)    0.9% (   0% -    2%) 0.000
          BrowseDayOfYearSSDVFacets        4.52     (12.3%)        4.57     (11.9%)    1.2% ( -20% -   28%) 0.754
                MedIntervalsOrdered      150.85      (5.1%)      152.78      (4.4%)    1.3% (  -7% -   11%) 0.400
                              range     2091.05      (5.3%)     2120.20      (5.9%)    1.4% (  -9% -   13%) 0.432
                        LowSpanNear      159.05      (1.5%)      161.43      (1.8%)    1.5% (  -1% -    4%) 0.004
                        MedSpanNear      199.13      (1.7%)      202.46      (1.8%)    1.7% (  -1% -    5%) 0.002
               HighIntervalsOrdered       37.62      (7.1%)       38.25      (5.7%)    1.7% ( -10% -   15%) 0.408
                      OrNotHighHigh      310.96      (7.1%)      316.57      (6.1%)    1.8% ( -10% -   16%) 0.389
                LowIntervalsOrdered      252.22      (6.2%)      256.93      (5.2%)    1.9% (  -8% -   14%) 0.303
                         HighPhrase      336.99      (1.8%)      344.07      (1.6%)    2.1% (  -1% -    5%) 0.000
                          LowPhrase      321.18      (1.2%)      328.13      (1.0%)    2.2% (   0% -    4%) 0.000
                        AndHighHigh      269.68      (3.2%)      275.84      (2.0%)    2.3% (  -2% -    7%) 0.007
                      OrHighNotHigh      398.46      (6.5%)      407.98      (5.2%)    2.4% (  -8% -   14%) 0.196
                           Wildcard      409.00      (2.5%)      419.40      (4.2%)    2.5% (  -4% -    9%) 0.021
                          MedPhrase      380.32      (1.4%)      390.52      (1.3%)    2.7% (   0% -    5%) 0.000
              BrowseMonthSSDVFacets        4.65     (15.9%)        4.82     (16.6%)    3.8% ( -24% -   43%) 0.465
                         OrHighHigh      313.96      (5.9%)      325.76      (3.3%)    3.8% (  -5% -   13%) 0.014
                          OrHighMed      470.47      (3.3%)      488.29      (2.0%)    3.8% (  -1% -    9%) 0.000
                            MedTerm      949.35      (4.9%)      986.95      (3.5%)    4.0% (  -4% -   13%) 0.003
                         AndHighMed      620.70      (1.6%)      647.41      (0.9%)    4.3% (   1% -    6%) 0.000
                  HighTermMonthSort     1388.47      (3.1%)     1450.18      (2.6%)    4.4% (  -1% -   10%) 0.000
                            Prefix3      547.58      (3.8%)      574.60      (5.4%)    4.9% (  -4% -   14%) 0.001
                       OrNotHighLow     1355.83      (2.9%)     1440.87      (2.8%)    6.3% (   0% -   12%) 0.000
                          OrHighLow      999.86      (2.1%)     1065.32      (1.9%)    6.5% (   2% -   10%) 0.000
                       OrHighNotMed     1269.64      (3.0%)     1354.01      (4.6%)    6.6% (   0% -   14%) 0.000
                           HighTerm     1125.78      (3.6%)     1206.11      (5.1%)    7.1% (  -1% -   16%) 0.000
                       OrHighNotLow     1078.83      (3.4%)     1162.69      (4.6%)    7.8% (   0% -   16%) 0.000
                            LowTerm     1725.12      (2.9%)     1868.98      (2.5%)    8.3% (   2% -   14%) 0.000
                  HighTermTitleSort      149.94      (2.8%)      164.35      (2.4%)    9.6% (   4% -   15%) 0.000
                         AndHighLow     1505.07      (3.0%)     1677.05      (4.0%)   11.4% (   4% -   19%) 0.000
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2732138703


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
       boolean confined,
       Function<IOContext, ReadAdvice> toReadAdvice) {
     assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+    AtomicInteger sharedPrefetchCounter = new AtomicInteger();

Review Comment:
   Thank you for the quick test case!
   
   Wow, so `isZeroOrPowerOfTwo(-1) = true`?  OK I see its javadocs:
   
   ```
     /**                                                                                                                                                                                                                                                                                                                                                                                           
      * Return true if, and only if, the provided integer - treated as an unsigned integer - is either                                                                                                                                                                                                                                                                                             
      * 0 or a power of two.                                                                                                                                                                                                                                                                                                                                                                       
      */
   ```
   
   -1 in binary 2s complement is `10000000000000000000000000000000` (1 followed by 31 zeros), which if you interpret as unsigned int is `2147483648` which is a power-of-two.  Tricky!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand merged PR #15585:
URL: https://github.com/apache/lucene/pull/15585


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2718691903


##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
       boolean confined,
       Function<IOContext, ReadAdvice> toReadAdvice) {
     assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+    AtomicInteger sharedPrefetchCounter = new AtomicInteger();

Review Comment:
   Is this creating one `AtomicInteger` per actually originally opened files?  And then each clone shares that same original `AtomicInteger`?
   
   Hmm what happens when we overflow the int?  What does `incrementAndGet` do then?  Silently overflow/wraparound to negative?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Added shared prefetch counter [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3820113225

   Thanks @uschindler and @dweiss.
   
   Actually, now I'm wondering about thread contention ... `AtomicInteger` isn't magic, it still  must obey the physics / mechanical sympathy down at the CPU core level.  OK I asked Claude Opus 4.5 about this: https://claude.ai/share/a9ddcc42-871c-4959-beac-c7b1a90bae0b
   
   Net/net it seems like we don't need to worry about that.  Yes, contention is bad ("MESI ping-pong!!"), and even more-so on ARM than recent x86-64 CPUs, but the amount of contention we're adding here should be tiny, and the crazy cost it is fixing (over-zealous `isLoaded()`) was much worse.  So we're good!  I'm trying to get nightlies working again so we can see if needles moved...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org