You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "shubhamsrkdev (via GitHub)" <gi...@apache.org> on 2026/01/16 22:27:36 UTC
[PR] Added shared prefetch counter [lucene]
shubhamsrkdev opened a new pull request, #15585:
URL: https://github.com/apache/lucene/pull/15585
### Problem
Across clone and slices of `MemorySegmentIndexInput` the consecutivePrefetchHitCount was getting reset leading to repeated calls.
### Solution
Added a shared prefetch counter, through reference - this should maintain count through clones and slices.
Fixes #15571
The benchmarks are looking good :
```
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value
IntSet 814.31 (4.5%) 794.78 (6.2%) -2.4% ( -12% - 8%) 0.159
TermDTSort 248.72 (5.8%) 245.11 (7.0%) -1.5% ( -13% - 12%) 0.476
LowSloppyPhrase 38.85 (3.9%) 38.37 (4.7%) -1.3% ( -9% - 7%) 0.360
BrowseDateSSDVFacets 0.90 (8.4%) 0.89 (9.0%) -0.8% ( -16% - 18%) 0.767
MedSloppyPhrase 45.58 (3.0%) 45.22 (3.4%) -0.8% ( -6% - 5%) 0.445
HighSloppyPhrase 12.27 (6.0%) 12.20 (5.9%) -0.6% ( -11% - 12%) 0.749
IntNRQ 295.84 (1.7%) 294.19 (1.7%) -0.6% ( -3% - 2%) 0.302
Respell 45.97 (2.5%) 45.77 (1.9%) -0.4% ( -4% - 4%) 0.539
BrowseMonthTaxoFacets 2.78 (2.0%) 2.77 (0.9%) -0.4% ( -3% - 2%) 0.455
HighSpanNear 27.73 (1.5%) 27.63 (1.4%) -0.3% ( -3% - 2%) 0.471
Fuzzy2 68.17 (2.6%) 67.95 (2.1%) -0.3% ( -4% - 4%) 0.668
HighIntervalsOrdered 32.46 (8.2%) 32.42 (9.3%) -0.1% ( -16% - 18%) 0.965
range 3603.08 (4.7%) 3600.78 (4.9%) -0.1% ( -9% - 9%) 0.966
BrowseRandomLabelSSDVFacets 3.22 (8.7%) 3.22 (8.1%) -0.0% ( -15% - 18%) 0.996
OrHighMedDayTaxoFacets 4.69 (1.9%) 4.69 (1.9%) 0.0% ( -3% - 3%) 0.975
HighTermDayOfYearSort 365.33 (1.4%) 365.41 (1.2%) 0.0% ( -2% - 2%) 0.959
Fuzzy1 70.44 (2.4%) 70.48 (1.8%) 0.1% ( -4% - 4%) 0.940
PKLookup 197.00 (1.7%) 197.17 (2.2%) 0.1% ( -3% - 4%) 0.893
BrowseRandomLabelTaxoFacets 2.39 (1.9%) 2.39 (2.0%) 0.2% ( -3% - 4%) 0.753
HighTermTitleBDVSort 63.91 (1.2%) 64.09 (1.0%) 0.3% ( -1% - 2%) 0.427
BrowseMonthSSDVFacets 4.52 (6.4%) 4.53 (7.2%) 0.3% ( -12% - 14%) 0.883
BrowseDayOfYearSSDVFacets 4.52 (8.1%) 4.53 (7.7%) 0.4% ( -14% - 17%) 0.888
MedIntervalsOrdered 61.93 (4.5%) 62.17 (5.3%) 0.4% ( -9% - 10%) 0.811
MedSpanNear 149.61 (1.3%) 150.36 (1.0%) 0.5% ( -1% - 2%) 0.167
LowSpanNear 102.70 (1.2%) 103.25 (0.8%) 0.5% ( -1% - 2%) 0.099
AndHighMedDayTaxoFacets 71.17 (0.8%) 71.56 (0.9%) 0.5% ( -1% - 2%) 0.053
LowIntervalsOrdered 273.02 (3.2%) 274.68 (4.2%) 0.6% ( -6% - 8%) 0.608
LowPhrase 54.15 (2.4%) 54.48 (1.7%) 0.6% ( -3% - 4%) 0.349
AndHighHighDayTaxoFacets 35.01 (0.9%) 35.28 (0.9%) 0.8% ( -1% - 2%) 0.006
MedTermDayTaxoFacets 35.19 (1.2%) 35.51 (0.8%) 0.9% ( -1% - 2%) 0.003
BrowseDayOfYearTaxoFacets 3.12 (4.1%) 3.15 (4.6%) 1.0% ( -7% - 10%) 0.480
BrowseDateTaxoFacets 3.10 (4.1%) 3.13 (4.7%) 1.0% ( -7% - 10%) 0.456
HighPhrase 87.16 (2.9%) 88.09 (1.9%) 1.1% ( -3% - 6%) 0.176
AndHighHigh 261.20 (2.9%) 266.49 (2.8%) 2.0% ( -3% - 7%) 0.025
MedPhrase 256.22 (1.8%) 261.49 (1.5%) 2.1% ( -1% - 5%) 0.000
HighTermMonthSort 1355.00 (2.3%) 1386.97 (3.3%) 2.4% ( -3% - 8%) 0.009
OrNotHighMed 585.83 (3.3%) 603.68 (3.9%) 3.0% ( -4% - 10%) 0.008
OrHighHigh 309.70 (5.2%) 319.44 (3.8%) 3.1% ( -5% - 12%) 0.029
Wildcard 815.29 (3.2%) 845.21 (2.9%) 3.7% ( -2% - 10%) 0.000
OrHighMed 626.92 (3.0%) 654.05 (2.1%) 4.3% ( 0% - 9%) 0.000
HighTermTitleSort 149.65 (2.8%) 156.38 (4.8%) 4.5% ( -3% - 12%) 0.000
MedTerm 1168.83 (4.1%) 1222.83 (5.0%) 4.6% ( -4% - 14%) 0.001
OrHighNotHigh 728.82 (3.9%) 762.67 (4.1%) 4.6% ( -3% - 13%) 0.000
OrNotHighHigh 867.08 (2.8%) 907.92 (3.7%) 4.7% ( -1% - 11%) 0.000
HighTerm 734.14 (5.8%) 769.02 (5.5%) 4.8% ( -6% - 16%) 0.008
OrHighNotMed 858.31 (3.1%) 901.62 (3.4%) 5.0% ( -1% - 11%) 0.000
AndHighMed 573.81 (2.6%) 603.39 (1.6%) 5.2% ( 0% - 9%) 0.000
OrNotHighLow 1102.50 (2.7%) 1165.73 (2.8%) 5.7% ( 0% - 11%) 0.000
LowTerm 1691.06 (3.5%) 1822.95 (3.7%) 7.8% ( 0% - 15%) 0.000
OrHighLow 1074.51 (2.6%) 1159.81 (2.5%) 7.9% ( 2% - 13%) 0.000
OrHighNotLow 1076.27 (4.0%) 1167.56 (5.4%) 8.5% ( 0% - 18%) 0.000
AndHighLow 1250.35 (2.1%) 1374.26 (3.3%) 9.9% ( 4% - 15%) 0.000
Prefix3 742.64 (16.9%) 861.01 (20.1%) 15.9% ( -18% - 63%) 0.007
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805845991
> I'll try to backport to 10.x soon ... something's wrong w/ my 10.x dev area .
D'oh! I was asking gradle to:
```
./gradlew clean check
```
(both at once) and that (weirdly) leads to the confusing deadlock. Why can't it just say "I'm sorry I cannot do both at once like that". Whereas if I separate them:
```
./gradlew clean
./gradlew check
```
'Tsall good!
I think this weird gradle bug only happens in `10.x` not `main` branch -- @rmuir thinks @dweiss fixed it already in `main`. But @rmuir suggested switching to the nuclear `git clean -xfd` option instead of trusting gradle for cleaning -- I'll try to do that from now on! Gotta erase my muscle memory...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3810828129
Hi,
> Thanks @shubhamsrkdev -- I think this is ready. I'll merge soon. Let's watch builds closely... I worry a bit about the GC implications of this change, for Lucene usage that has a great many clones (many queries in flight, GC being lazy since there's lots of heap headroom say) and all of those clones pointing to this one `AtomicInteger`... likely it's fine but we've had GC issues in the past when `MMapDirectory` tried to carefully track clones (before Panama gave us better mmap APIs -- no more scary `Unsafe` for unmapping).
this is not an issue here. The clones only share the same AtomicInteger. Basically all clones also refer to the same Arena (which is grouped, too so has additional stuff). Basically this is all fine, the problems with tracking clones were a different category of issues because we needed a pointer from the parent input to its childs and this goes worse. Here we just refer to some same object, which itsself has no reference to the clones. Once refcounting is zero the atomic integer gets free for GC. The problem with the master refering to childs was that as ong as the master input refers to its childs, those cannot be GCed. As we never call close on clones there is no way to figure out for the master that it is gone. The hard relationship master -> clone was the killer.
Uwe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704668163
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
* chunkSizePower</code>).
*/
abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+ /** Shared counter for prefetch hit tracking. */
+ static final class SharedPrefetchCounter {
+ private int count;
Review Comment:
Should we make count `volatile` to ensure visibility across threads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3790523874
> Which corpus was the luceneutil benchy results run on? Crazy how fast Lucene queries have become!
This was on `wikimediumall` . I will mention this in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721591904
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
boolean confined,
Function<IOContext, ReadAdvice> toReadAdvice) {
assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+ AtomicInteger sharedPrefetchCounter = new AtomicInteger();
Review Comment:
>Is this creating one AtomicInteger per actually originally opened files? And then each clone shares that same original AtomicInteger?
Yes!
>Hmm what happens when we overflow the int? What does incrementAndGet do then? Silently overflow/wraparound to negative?
It does wraparound after `Integer.MAX_VALUE` to Integer.MIN_VALUE and subsequent negative values. But it would just cause an extra prefetch because of `isZeroOrPowerOfTwo` ?`isZeroOrPowerOfTwo(Integer.MIN_VALUE)` would give true but subsequent increments would just give false as expected. For example:
```
import java.util.concurrent.atomic.AtomicInteger;
class Main {
public static void main(String[] args) {
AtomicInteger counter = new AtomicInteger(Integer.MAX_VALUE);
while(counter.incrementAndGet() < 0)
{
System.out.println(isZeroOrPowerOfTwo(counter.get()));
counter = new AtomicInteger(counter.incrementAndGet());
}
}
public static boolean isZeroOrPowerOfTwo(int x) {
return (x & (x - 1)) == 0;
}
}
```
This would give:
```
true
false
false
false
false
false
false
false
false
false
false
.
.
.
.
.
.
.
.
.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3819258882
> (both at once) and that (weirdly) leads to the confusing deadlock. Why can't it just say "I'm sorry I cannot do both at once like that".
It's complicated - there are included builds (composite builds) and they're treated differently... chicken and egg.
> But @rmuir suggested switching to the nuclear `git clean -xfd` option instead of trusting gradle for cleaning
It's also much faster... My choice too, really.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2718698084
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -316,7 +335,7 @@ public void prefetch(long offset, long length) throws IOException {
ensureOpen();
- if (BitUtil.isZeroOrPowerOfTwo(consecutivePrefetchHitCount++) == false) {
+ if (BitUtil.isZeroOrPowerOfTwo(sharedPrefetchCounter.incrementAndGet()) == false) {
Review Comment:
Does it matter that before it was get-and-increment and now it's increment-and-get? E.g. we will now start with 0 and get `true` the first time, but previously false, so won't we fail to check the first time we come through with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721595114
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -316,7 +335,7 @@ public void prefetch(long offset, long length) throws IOException {
ensureOpen();
- if (BitUtil.isZeroOrPowerOfTwo(consecutivePrefetchHitCount++) == false) {
+ if (BitUtil.isZeroOrPowerOfTwo(sharedPrefetchCounter.incrementAndGet()) == false) {
Review Comment:
Good catch! I can change this to `getAndIncrement()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3786754721
Which corpus was the luceneutil benchy results run on? Crazy how fast Lucene queries have become!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805698102
I'll try to backport to 10.x soon ... something's wrong w/ my 10.x dev area ... I think this is gradle solving the halting problem (confessing it is stuck / will never halt) for Lucene's gradle config, and reporting it in a very unintuitive manner :) ... I'll try a fresh clone (I did kill its daemons already):
```
> Task :missing-doclet:jar
Unable to make progress running work. The following items are queued for execution but none of them can be started:
- Build ':':
- Waiting for nodes:
- :lucene:spatial3d:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:spatial3d:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:spatial3d:test (SHOULD_RUN)] )
- :lucene:analysis.tests:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:analysis.tests:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:analysis.tests:test (SHOULD_RUN)] )
- :lucene:memory:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:memory:ecjLint (SHOULD_RUN), :lucene:memory:forbiddenApis (SHOULD_RUN), :lucene:memory:licenses (SHOULD_RUN), :lucene:memory:spotlessCheck (SHOULD_RUN), :lucene:memory:test (SHOULD_RUN), :lucene:memory:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:m\
emory:licenses (SHOULD_RUN), :lucene:memory:ecjLint (SHOULD_RUN), :lucene:memory:forbiddenApis (SHOULD_RUN), :lucene:memory:validateSourcePatterns (SHOULD_RUN), :lucene:memory:test (SHOULD_RUN), :lucene:memory:spotlessCheck (SHOULD_RUN)], has-failed-dependency=false )
- :lucene:backward-codecs:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:backward-codecs:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:backward-codecs:test (SHOULD_RUN)] )
- :lucene:analysis:icu:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:analysis:icu:ecjLint (SHOULD_RUN), :lucene:analysis:icu:forbiddenApis (SHOULD_RUN), :lucene:analysis:icu:genRbbiChecksumCheck (EXECUTED), :lucene:analysis:icu:genUtr30DataFilesChecksumCheck (EXECUTED), :lucene:analysis:icu:licenses (SHOULD_RUN), :lucene:an\
alysis:icu:spotlessCheck (SHOULD_RUN), :lucene:analysis:icu:test (SHOULD_RUN), :lucene:analysis:icu:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:analysis:icu:spotlessCheck (SHOULD_RUN), :lucene:analysis:icu:validateSourcePatterns (SHOULD_RUN), :lucene:analysis:icu:forbiddenApis (SHOULD_RUN), :lucene:analysis:icu:test (SHOULD_RUN), :lucene:analysis:icu:ecjLint (SHOULD\
_RUN), :lucene:analysis:icu:licenses (SHOULD_RUN)], has-failed-dependency=false )
- :lucene:queryparser:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:queryparser:ecjLint (SHOULD_RUN), :lucene:queryparser:forbiddenApis (SHOULD_RUN), :lucene:queryparser:javaccParserClassicChecksumCheck (EXECUTED), :lucene:queryparser:javaccParserFlexibleChecksumCheck (EXECUTED), :lucene:queryparser:javaccParserSurroundChec\
ksumCheck (EXECUTED), :lucene:queryparser:licenses (SHOULD_RUN), :lucene:queryparser:spotlessCheck (SHOULD_RUN), :lucene:queryparser:test (SHOULD_RUN), :lucene:queryparser:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:queryparser:forbiddenApis (SHOULD_RUN), :lucene:queryparser:validateSourcePatterns (SHOULD_RUN), :lucene:queryparser:ecjLint (SHOULD_RUN), :lucene:query\
parser:licenses (SHOULD_RUN), :lucene:queryparser:spotlessCheck (SHOULD_RUN), :lucene:queryparser:test (SHOULD_RUN)], has-failed-dependency=false )
- :lucene:join:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:join:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:join:test (SHOULD_RUN)] )
- :lucene:analysis:morfologik:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, dependencies=[:lucene:analysis:morfologik:ecjLint (SHOULD_RUN), :lucene:analysis:morfologik:forbiddenApis (SHOULD_RUN), :lucene:analysis:morfologik:licenses (SHOULD_RUN), :lucene:analysis:morfologik:spotlessCheck (SHOULD_RUN), :lucene:analysis:morfologik:test (SHOULD_RUN\
), :lucene:analysis:morfologik:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:analysis:morfologik:spotlessCheck (SHOULD_RUN), :lucene:analysis:morfologik:validateSourcePatterns (SHOULD_RUN), :lucene:analysis:morfologik:licenses (SHOULD_RUN), :lucene:analysis:morfologik:test (SHOULD_RUN), :lucene:analysis:morfologik:ecjLint (SHOULD_RUN), :lucene:analysis:morfologik:forb\
iddenApis (SHOULD_RUN)], has-failed-dependency=false )
- :lucene:highlighter:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:highlighter:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:highlighter:test (SHOULD_RUN)] )
- :lucene:analysis:morfologik:wipeTaskTemp (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=finalizer :lucene:analysis:morfologik:wipeTaskTemp ordinal: task group 1, delegate: default group, no dependencies, finalizes=[:lucene:analysis:morfologik:test (SHOULD_RUN)] )
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2721591904
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
boolean confined,
Function<IOContext, ReadAdvice> toReadAdvice) {
assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+ AtomicInteger sharedPrefetchCounter = new AtomicInteger();
Review Comment:
>Is this creating one AtomicInteger per actually originally opened files? And then each clone shares that same original AtomicInteger?
Yes!
>Hmm what happens when we overflow the int? What does incrementAndGet do then? Silently overflow/wraparound to negative?
It does wraparound after `Integer.MAX_VALUE` to `Integer.MIN_VALUE` and subsequent negative values. But it would just cause an extra prefetch because of `isZeroOrPowerOfTwo` ?`isZeroOrPowerOfTwo(Integer.MIN_VALUE)` would give true but subsequent increments would just give false as expected. For example:
```
import java.util.concurrent.atomic.AtomicInteger;
class Main {
public static void main(String[] args) {
AtomicInteger counter = new AtomicInteger(Integer.MAX_VALUE);
while(counter.incrementAndGet() < 0)
{
System.out.println(isZeroOrPowerOfTwo(counter.get()));
}
}
public static boolean isZeroOrPowerOfTwo(int x) {
return (x & (x - 1)) == 0;
}
}
```
This would give:
```
true
false
false
false
false
false
false
false
false
false
false
.
.
.
.
.
.
.
.
.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3810843016
Basically everything is shared between master and clone (including the memory segments, and their arena - implicit) - the clones and master only differ in their position and slice coordinates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2705127972
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
* chunkSizePower</code>).
*/
abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+ /** Shared counter for prefetch hit tracking. */
+ static final class SharedPrefetchCounter {
+ private int count;
Review Comment:
I suggested volatile because I don't think the counter needs to be perfectly precise, and using thread-safe types like AtomicInteger or LongAdder might increase thread contention. That said, perhaps we could test all three options (current implementation, volatile, AtomicInteger) and choose the fastest one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704817314
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
* chunkSizePower</code>).
*/
abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+ /** Shared counter for prefetch hit tracking. */
+ static final class SharedPrefetchCounter {
+ private int count;
Review Comment:
or LongAdder which is designed exactly for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2704974447
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
* chunkSizePower</code>).
*/
abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+ /** Shared counter for prefetch hit tracking. */
+ static final class SharedPrefetchCounter {
+ private int count;
Review Comment:
Maybe I can use `AtomicInteger` ? Should cover thread safety... `LongAdder` doesn't have an increment and get, maybe it is better for less read intensive scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2732151120
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
boolean confined,
Function<IOContext, ReadAdvice> toReadAdvice) {
assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+ AtomicInteger sharedPrefetchCounter = new AtomicInteger();
Review Comment:
So net/net I think the behavior is reasonable -- we check with 1/2 the frequency, up until `1 << 31 = 2147483648` at which point we stop halving the frequency of checks and do it every `2147483648` calls. That's great -- no exception, still checking (rarely!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3805633205
> The benchmarks are looking good :
If I understand correctly, Lucene `main` now defaults to no IOContext-based `madvise`? And luceneutil just goes with Lucene's defaults here? Which makes these benchmark results very surprising! Oh, wait!, that is a different `madvise`.
This change is about Lucene's dynamic prefetch -- `madvise(WILL_NEED)` for specific byte ranges of the file. (Lucene also has static prefetch -- up front `madvise(WILL_NEED)` on the entire file when it's opened, but we are not changing that here). So this change fixes the frequency of clones calling `.isLoaded()` on the entire file. In the hot case, this is disastrous (slowest call for `.isLoaded()`).
Still, for `HighTerm`, which is a high frequency term, we create the clones (once per segment) and then use them for a long-ish time. I would've thought that the high `.isLoaded()` cost would've been amortized and we wouldn't see such gains from `HighTerm`. Confusing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "shubhamsrkdev (via GitHub)" <gi...@apache.org>.
shubhamsrkdev commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2705209143
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -40,6 +40,22 @@
* chunkSizePower</code>).
*/
abstract class MemorySegmentIndexInput extends IndexInput implements MemorySegmentAccessInput {
+
+ /** Shared counter for prefetch hit tracking. */
+ static final class SharedPrefetchCounter {
+ private int count;
Review Comment:
Yes, I did that - looks like they are almost the same `AtomicInteger` being somewhat better?
- `volatile`
```
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value
BrowseDayOfYearTaxoFacets 3.28 (9.4%) 3.20 (4.8%) -2.6% ( -15% - 12%) 0.280
BrowseDateTaxoFacets 3.24 (7.9%) 3.18 (4.9%) -2.0% ( -13% - 11%) 0.331
TermDTSort 261.53 (3.2%) 259.46 (3.5%) -0.8% ( -7% - 6%) 0.457
range 3676.85 (4.3%) 3660.49 (5.1%) -0.4% ( -9% - 9%) 0.764
BrowseRandomLabelTaxoFacets 2.40 (4.3%) 2.39 (2.0%) -0.4% ( -6% - 6%) 0.691
MedIntervalsOrdered 147.92 (6.6%) 147.43 (6.4%) -0.3% ( -12% - 13%) 0.872
HighIntervalsOrdered 17.40 (8.7%) 17.35 (9.0%) -0.3% ( -16% - 19%) 0.914
OrHighNotHigh 371.74 (5.7%) 370.95 (6.3%) -0.2% ( -11% - 12%) 0.911
OrNotHighHigh 266.13 (6.5%) 265.65 (6.8%) -0.2% ( -12% - 13%) 0.931
IntNRQ 466.76 (1.9%) 466.10 (1.9%) -0.1% ( -3% - 3%) 0.812
HighSpanNear 34.12 (3.3%) 34.11 (4.6%) -0.0% ( -7% - 8%) 0.991
LowIntervalsOrdered 116.50 (2.5%) 116.49 (2.5%) -0.0% ( -4% - 5%) 0.991
OrHighMedDayTaxoFacets 7.60 (1.9%) 7.60 (1.9%) 0.1% ( -3% - 3%) 0.860
HighTermTitleBDVSort 31.69 (2.3%) 31.80 (2.2%) 0.3% ( -4% - 4%) 0.625
BrowseMonthTaxoFacets 2.75 (1.7%) 2.76 (0.3%) 0.4% ( -1% - 2%) 0.356
AndHighHighDayTaxoFacets 29.66 (1.4%) 29.77 (1.2%) 0.4% ( -2% - 2%) 0.387
HighSloppyPhrase 35.65 (2.0%) 35.78 (2.6%) 0.4% ( -4% - 5%) 0.625
LowSloppyPhrase 117.98 (2.5%) 118.53 (3.2%) 0.5% ( -5% - 6%) 0.611
PKLookup 198.36 (2.2%) 199.34 (1.7%) 0.5% ( -3% - 4%) 0.424
MedSloppyPhrase 52.11 (2.3%) 52.37 (2.9%) 0.5% ( -4% - 5%) 0.542
BrowseMonthSSDVFacets 4.65 (14.6%) 4.67 (15.4%) 0.5% ( -25% - 35%) 0.915
HighTermDayOfYearSort 354.00 (1.7%) 356.01 (1.4%) 0.6% ( -2% - 3%) 0.251
MedSpanNear 118.13 (2.8%) 118.85 (3.4%) 0.6% ( -5% - 7%) 0.540
MedTermDayTaxoFacets 17.07 (2.1%) 17.18 (1.7%) 0.6% ( -3% - 4%) 0.297
Respell 40.13 (1.8%) 40.38 (1.8%) 0.6% ( -2% - 4%) 0.266
AndHighMedDayTaxoFacets 29.22 (1.1%) 29.43 (1.2%) 0.7% ( -1% - 3%) 0.051
HighPhrase 54.55 (1.4%) 54.95 (1.5%) 0.7% ( -2% - 3%) 0.107
Prefix3 149.74 (5.4%) 151.03 (3.4%) 0.9% ( -7% - 10%) 0.549
MedPhrase 155.10 (1.2%) 156.71 (1.4%) 1.0% ( -1% - 3%) 0.013
LowSpanNear 168.63 (3.2%) 170.49 (3.9%) 1.1% ( -5% - 8%) 0.324
Fuzzy1 80.40 (3.3%) 81.62 (2.9%) 1.5% ( -4% - 7%) 0.123
Fuzzy2 78.61 (3.9%) 79.84 (3.2%) 1.6% ( -5% - 8%) 0.163
OrNotHighMed 542.60 (4.8%) 551.16 (4.2%) 1.6% ( -7% - 11%) 0.266
IntSet 794.07 (3.9%) 806.99 (3.1%) 1.6% ( -5% - 8%) 0.142
HighTermTitleSort 129.70 (2.8%) 131.85 (2.8%) 1.7% ( -3% - 7%) 0.059
AndHighHigh 345.95 (3.0%) 352.27 (5.9%) 1.8% ( -6% - 10%) 0.214
BrowseDayOfYearSSDVFacets 4.63 (14.3%) 4.71 (16.4%) 1.9% ( -25% - 38%) 0.694
OrHighHigh 407.65 (3.1%) 416.03 (5.8%) 2.1% ( -6% - 11%) 0.161
OrHighNotMed 789.81 (4.1%) 806.23 (6.1%) 2.1% ( -7% - 12%) 0.206
BrowseRandomLabelSSDVFacets 3.09 (4.5%) 3.17 (9.7%) 2.5% ( -11% - 17%) 0.305
BrowseDateSSDVFacets 0.91 (7.8%) 0.93 (5.8%) 2.5% ( -10% - 17%) 0.244
OrHighMed 579.17 (2.2%) 598.98 (4.3%) 3.4% ( -3% - 10%) 0.002
AndHighMed 529.81 (1.6%) 548.32 (3.6%) 3.5% ( -1% - 8%) 0.000
HighTermMonthSort 1362.64 (2.9%) 1420.03 (2.8%) 4.2% ( -1% - 10%) 0.000
Wildcard 788.01 (3.1%) 821.42 (3.4%) 4.2% ( -2% - 11%) 0.000
LowPhrase 572.79 (1.4%) 599.26 (1.3%) 4.6% ( 1% - 7%) 0.000
HighTerm 1037.42 (2.5%) 1085.74 (6.2%) 4.7% ( -3% - 13%) 0.002
MedTerm 1140.70 (3.7%) 1198.80 (5.0%) 5.1% ( -3% - 14%) 0.000
OrHighNotLow 1093.24 (3.5%) 1157.63 (4.9%) 5.9% ( -2% - 14%) 0.000
LowTerm 1595.88 (2.6%) 1714.75 (7.5%) 7.4% ( -2% - 18%) 0.000
OrHighLow 1027.76 (2.5%) 1107.88 (2.1%) 7.8% ( 3% - 12%) 0.000
OrNotHighLow 1456.36 (2.5%) 1575.86 (2.5%) 8.2% ( 3% - 13%) 0.000
AndHighLow 1279.96 (3.2%) 1418.90 (3.5%) 10.9% ( 3% - 18%) 0.000
```
- `AtomicInteger`
```
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value
IntSet 824.23 (5.1%) 812.38 (5.1%) -1.4% ( -11% - 9%) 0.371
Fuzzy2 79.65 (3.1%) 78.82 (3.7%) -1.0% ( -7% - 5%) 0.335
Fuzzy1 71.11 (2.6%) 70.48 (2.9%) -0.9% ( -6% - 4%) 0.312
MedTermDayTaxoFacets 40.99 (1.1%) 40.74 (1.1%) -0.6% ( -2% - 1%) 0.073
TermDTSort 255.15 (3.2%) 253.86 (2.6%) -0.5% ( -6% - 5%) 0.588
Respell 33.25 (1.7%) 33.09 (2.3%) -0.5% ( -4% - 3%) 0.437
AndHighHighDayTaxoFacets 16.63 (1.4%) 16.57 (1.7%) -0.4% ( -3% - 2%) 0.429
IntNRQ 438.66 (1.9%) 437.29 (1.4%) -0.3% ( -3% - 3%) 0.557
LowSloppyPhrase 33.33 (4.3%) 33.32 (4.9%) -0.0% ( -8% - 9%) 0.980
HighTermTitleBDVSort 74.23 (1.5%) 74.24 (1.7%) 0.0% ( -3% - 3%) 0.973
BrowseMonthTaxoFacets 2.77 (1.5%) 2.77 (1.5%) 0.0% ( -2% - 3%) 0.961
BrowseDateSSDVFacets 0.87 (10.0%) 0.87 (9.3%) 0.1% ( -17% - 21%) 0.985
PKLookup 198.31 (1.9%) 198.66 (2.3%) 0.2% ( -3% - 4%) 0.789
HighTermDayOfYearSort 293.62 (1.8%) 294.17 (2.2%) 0.2% ( -3% - 4%) 0.767
BrowseDateTaxoFacets 3.15 (4.8%) 3.15 (4.7%) 0.2% ( -8% - 10%) 0.872
BrowseDayOfYearTaxoFacets 3.17 (4.7%) 3.18 (4.6%) 0.3% ( -8% - 10%) 0.862
OrHighMedDayTaxoFacets 11.82 (2.4%) 11.85 (1.9%) 0.3% ( -3% - 4%) 0.702
HighSloppyPhrase 25.82 (3.4%) 25.91 (4.3%) 0.3% ( -7% - 8%) 0.788
BrowseRandomLabelTaxoFacets 2.38 (1.7%) 2.39 (1.9%) 0.3% ( -3% - 3%) 0.540
BrowseRandomLabelSSDVFacets 3.22 (11.0%) 3.24 (10.4%) 0.7% ( -18% - 24%) 0.842
OrNotHighMed 402.90 (6.4%) 405.86 (6.8%) 0.7% ( -11% - 14%) 0.726
HighSpanNear 41.21 (2.7%) 41.54 (2.6%) 0.8% ( -4% - 6%) 0.345
MedSloppyPhrase 120.35 (2.8%) 121.37 (3.4%) 0.8% ( -5% - 7%) 0.400
AndHighMedDayTaxoFacets 102.67 (0.7%) 103.56 (0.7%) 0.9% ( 0% - 2%) 0.000
BrowseDayOfYearSSDVFacets 4.52 (12.3%) 4.57 (11.9%) 1.2% ( -20% - 28%) 0.754
MedIntervalsOrdered 150.85 (5.1%) 152.78 (4.4%) 1.3% ( -7% - 11%) 0.400
range 2091.05 (5.3%) 2120.20 (5.9%) 1.4% ( -9% - 13%) 0.432
LowSpanNear 159.05 (1.5%) 161.43 (1.8%) 1.5% ( -1% - 4%) 0.004
MedSpanNear 199.13 (1.7%) 202.46 (1.8%) 1.7% ( -1% - 5%) 0.002
HighIntervalsOrdered 37.62 (7.1%) 38.25 (5.7%) 1.7% ( -10% - 15%) 0.408
OrNotHighHigh 310.96 (7.1%) 316.57 (6.1%) 1.8% ( -10% - 16%) 0.389
LowIntervalsOrdered 252.22 (6.2%) 256.93 (5.2%) 1.9% ( -8% - 14%) 0.303
HighPhrase 336.99 (1.8%) 344.07 (1.6%) 2.1% ( -1% - 5%) 0.000
LowPhrase 321.18 (1.2%) 328.13 (1.0%) 2.2% ( 0% - 4%) 0.000
AndHighHigh 269.68 (3.2%) 275.84 (2.0%) 2.3% ( -2% - 7%) 0.007
OrHighNotHigh 398.46 (6.5%) 407.98 (5.2%) 2.4% ( -8% - 14%) 0.196
Wildcard 409.00 (2.5%) 419.40 (4.2%) 2.5% ( -4% - 9%) 0.021
MedPhrase 380.32 (1.4%) 390.52 (1.3%) 2.7% ( 0% - 5%) 0.000
BrowseMonthSSDVFacets 4.65 (15.9%) 4.82 (16.6%) 3.8% ( -24% - 43%) 0.465
OrHighHigh 313.96 (5.9%) 325.76 (3.3%) 3.8% ( -5% - 13%) 0.014
OrHighMed 470.47 (3.3%) 488.29 (2.0%) 3.8% ( -1% - 9%) 0.000
MedTerm 949.35 (4.9%) 986.95 (3.5%) 4.0% ( -4% - 13%) 0.003
AndHighMed 620.70 (1.6%) 647.41 (0.9%) 4.3% ( 1% - 6%) 0.000
HighTermMonthSort 1388.47 (3.1%) 1450.18 (2.6%) 4.4% ( -1% - 10%) 0.000
Prefix3 547.58 (3.8%) 574.60 (5.4%) 4.9% ( -4% - 14%) 0.001
OrNotHighLow 1355.83 (2.9%) 1440.87 (2.8%) 6.3% ( 0% - 12%) 0.000
OrHighLow 999.86 (2.1%) 1065.32 (1.9%) 6.5% ( 2% - 10%) 0.000
OrHighNotMed 1269.64 (3.0%) 1354.01 (4.6%) 6.6% ( 0% - 14%) 0.000
HighTerm 1125.78 (3.6%) 1206.11 (5.1%) 7.1% ( -1% - 16%) 0.000
OrHighNotLow 1078.83 (3.4%) 1162.69 (4.6%) 7.8% ( 0% - 16%) 0.000
LowTerm 1725.12 (2.9%) 1868.98 (2.5%) 8.3% ( 2% - 14%) 0.000
HighTermTitleSort 149.94 (2.8%) 164.35 (2.4%) 9.6% ( 4% - 15%) 0.000
AndHighLow 1505.07 (3.0%) 1677.05 (4.0%) 11.4% ( 4% - 19%) 0.000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2732138703
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
boolean confined,
Function<IOContext, ReadAdvice> toReadAdvice) {
assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+ AtomicInteger sharedPrefetchCounter = new AtomicInteger();
Review Comment:
Thank you for the quick test case!
Wow, so `isZeroOrPowerOfTwo(-1) = true`? OK I see its javadocs:
```
/**
* Return true if, and only if, the provided integer - treated as an unsigned integer - is either
* 0 or a power of two.
*/
```
-1 in binary 2s complement is `10000000000000000000000000000000` (1 followed by 31 zeros), which if you interpret as unsigned int is `2147483648` which is a power-of-two. Tricky!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand merged PR #15585:
URL: https://github.com/apache/lucene/pull/15585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #15585:
URL: https://github.com/apache/lucene/pull/15585#discussion_r2718691903
##########
lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java:
##########
@@ -74,12 +75,28 @@ public static MemorySegmentIndexInput newInstance(
boolean confined,
Function<IOContext, ReadAdvice> toReadAdvice) {
assert Arrays.stream(segments).map(MemorySegment::scope).allMatch(arena.scope()::equals);
+ AtomicInteger sharedPrefetchCounter = new AtomicInteger();
Review Comment:
Is this creating one `AtomicInteger` per actually originally opened files? And then each clone shares that same original `AtomicInteger`?
Hmm what happens when we overflow the int? What does `incrementAndGet` do then? Silently overflow/wraparound to negative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Added shared prefetch counter [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15585:
URL: https://github.com/apache/lucene/pull/15585#issuecomment-3820113225
Thanks @uschindler and @dweiss.
Actually, now I'm wondering about thread contention ... `AtomicInteger` isn't magic, it still must obey the physics / mechanical sympathy down at the CPU core level. OK I asked Claude Opus 4.5 about this: https://claude.ai/share/a9ddcc42-871c-4959-beac-c7b1a90bae0b
Net/net it seems like we don't need to worry about that. Yes, contention is bad ("MESI ping-pong!!"), and even more-so on ARM than recent x86-64 CPUs, but the amount of contention we're adding here should be tiny, and the crazy cost it is fixing (over-zealous `isLoaded()`) was much worse. So we're good! I'm trying to get nightlies working again so we can see if needles moved...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org