You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "neoremind (via GitHub)" <gi...@apache.org> on 2026/05/25 06:33:08 UTC
[PR] Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead [lucene]
neoremind opened a new pull request, #16119:
URL: https://github.com/apache/lucene/pull/16119
Inspired by #13543 for `write(int)`, I was profiling the write path and noticed a similar optimization can be applied to `writeBytes`, this can eliminate unnecessary `growIfNeeded` checks and the large-byte-array bypass checks.
**Why this is safe:**
1. `XBufferedOutputStream` uses a fixed 8KB buffer, it never resizes, `growIfNeeded` is pure overhead.
2. The fallback to `super.write(b, off, len)` retains the existing behavior for large byte arrays that should bypass buffering entirely and chunk them later for real I/O.
## Verification
Since Lucene indexing is CPU-intensive and the I/O path isn't the time-consuming part, there is no significant processing time difference in luceneutil nightly benchmarks. But I did profiling with JFR and verified the overhead is completely removed.
Note that I can only sample `growIfNeeded` since it's a method call, the large-byte-array bypass check is inlined and hard to measure, so the real benefit should be slightly better than what the numbers below show.
### Wikipedia medium
33M docs, single-thread, with dv, wait for commit/merge, use CMS and TieredMergePolicy
| | growIfNeeded samples | Total CPU samples | Ratio |
|---|---|---|---|
| **Baseline** | 26 | 409,250 | 0.01% |
See Appendix 2. for detailed call stack captured.
[flamegraph link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/bench-index-bench_medium-wikimediumall.lucene.baseline.Lucene104.nd33.3326M-2026.05.23.14.42.03-baseline.html)
[log link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/bench_medium.wikimediumall.lucene.baseline.Lucene104.nd33.3326M.log)
What to call out is that `writeBytes` is not the dominanting write op in `OutputStreamIndexOutput`, only 5% of total calls. But the savings are still positive. In a 33M wikipedia indexing, it removes 245M unnecessary `growIfNeeded` and large-byte-array bypass checks.
I intercepted `OutputStreamIndexOutput` to count every write call (see how-to in Appendix 1). The stats show `writeInt` dominates by call count (61.4%), and `writeBytes` handles nearly half the total bytes despite being only 5% of calls. And 93% of `writeBytes` calls are less than 128 bytes, they all hit the optimization.
```
WriteStats: totalOps=4,875,075,375 totalBytes=26,174,155,590
writeByte: 1,627,290,133 ( 33.4% calls, 6.2% bytes)
writeBytes: 245,247,290 ( 5.0% calls, 47.9% bytes, avgLen=51)
writeInt: 2,991,534,308 ( 61.4% calls, 45.7% bytes)
writeLong: 5,474,411 ( 0.1% calls, 0.2% bytes)
writeShort: 5,529,233 ( 0.1% calls, 0.0% bytes)
writeBytes size distribution:
[1-16]: 164,778,837
[17-128]: 65,456,422
[129-1K]: 14,993,877
[1K-8K]: 1,722
[>8K]: 16,432
```
### KNN indexing
1M 1024-dim float32 vectors, wait for commit/merge, use CMS, force merge at last
| | growIfNeeded samples | Total CPU samples | Ratio |
|---|---|---|---|
| **Baseline** | 4 | 1,943,015 | 0.0002% |
See Appendix 2. for detailed call stack captured.
[flamegraph link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/knn-perf-test.2026-05-22-before.html)
[log link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/knnPerfTest.py_20260522_070450.log)
For knn, `writeBytes` dominates at 78.4% of all write calls, indexing 1M vectors involves tens of millions of `writeBytes` ops, and all of them benefit from the fast path. But since knn is super CPU-intensive with vectorized dot product dominating the profiling, the small improvement diminishes as it's barely visible in the samples, but still a micro-level win.
```
During multi-thread indexing with 1024-dim float32 vector, rerank to improve precision. Wait for commit and merge to complete:
WriteStats: totalOps=55,596,839 totalBytes=12,931,045,866
writeByte: 11,985,442 ( 21.6% calls, 0.1% bytes)
writeBytes: 43,597,865 ( 78.4% calls, 99.9% bytes, avgLen=296)
writeInt: 4,752 ( 0.0% calls, 0.0% bytes)
writeLong: 8,574 ( 0.0% calls, 0.0% bytes)
writeShort: 206 ( 0.0% calls, 0.0% bytes)
writeBytes size distribution:
[1-16]: 40,481,940
[17-128]: 7,426
[129-1K]: 32,581
[1K-8K]: 3,075,918
[>8K]: 0
Follow-up merge to 1 segment:
WriteStats: totalOps=21,901,875 totalBytes=4,252,427,320
writeByte: 4,719,616 ( 21.5% calls, 0.1% bytes)
writeBytes: 17,180,254 ( 78.4% calls, 99.9% bytes, avgLen=247)
writeInt: 38 ( 0.0% calls, 0.0% bytes)
writeLong: 1,966 ( 0.0% calls, 0.0% bytes)
writeShort: 1 ( 0.0% calls, 0.0% bytes)
writeBytes size distribution:
[1-16]: 16,169,470
[17-128]: 478
[129-1K]: 9,791
[1K-8K]: 1,000,515
[>8K]: 0
```
## Write path deep dive
`FSDirectory.FSIndexOutput` writes data through a layered stack:
```
Lucene write call (writeByte, writeBytes, writeShort, writeInt, writeLong)
→ XBufferedOutputStream (heap byte[] buffer, 8KB)
→ CheckedOutputStream (CRC32 checksum, inline on flush)
→ FilterOutputStream (chunks writes to ≤ CHUNK_SIZE)
→ ChannelOutputStream
→ FileChannelImpl.write(HeapByteBuffer)
→ sun.nio.ch.IOUtil.write()
→ getTemporaryDirectBuffer() (thread-local cached DirectByteBuffer)
→ memcpy heap → direct
→ pwrite() syscall (kernel page cache → disk)
```
For the common case that small writes that fit in the buffer, do `System.arraycopy`, no unnecessary checks. For large arrays that the buffer can't hold, fall to `BufferedOutputStream.write()` which flushes the buffer and writes directly to the underlying stream, the `FilterOutputStream` in `FSDirectory` then chunks those large writes to less than 8K before hitting the file channel, preventing unwanted larger native buffer allocation. This is true, so I've also updated the stale `CHUNK_SIZE` comment as the original comment is a bit misleading.
<img width="2777" height="3326" alt="wite-path-deep-dive" src="https://github.com/user-attachments/assets/e5e84a3c-a95d-4203-a9ff-d43e1c3f2341" />
## Appendix 1. Statistical analysis method of lucene write path
Write statistics were captured using a modified luceneutil that intercepts all write calls and performs statistical analysis after indexing completes: [luceneutil commit](https://github.com/neoremind/luceneutil/commit/e35084338e2e45f004310f1d8b0f3f4492e76349#diff-9a2d54be8c91660fcf380e6df9944630a045021e5045a05754e1534277296714).
## Appendix 2. Detailed call stacks hitting growIfNeeded captured by JFR
<details>
<summary>Baseline top stacks of wikipedia 33MM</summary>
--- Count: 3 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 944
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 636
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 926
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 666
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress() line: 194
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush() line: 259
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument() line: 191
org.apache.lucene.index.StoredFieldsConsumer.finishDocument() line: 94
org.apache.lucene.index.IndexingChain.finishStoredFields() line: 565
org.apache.lucene.index.IndexingChain.processDocument() line: 657
org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments() line: 291
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 428
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
Total CPU samples: 409250
Matched: 26
Percentage: 0.01%
</details>
<details>
<summary>Baseline top stacks of knn 1MM</summary>
--- Count: 3 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173
org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1
org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144
org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272
org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 159
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173
org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1
org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144
org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272
org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 159
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726
Total CPU samples: 1943015
Matched: 4
Percentage: 0.00%
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead [lucene]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on PR #16119:
URL: https://github.com/apache/lucene/pull/16119#issuecomment-6328934245
This never actually got back ported to 10.5, I will back port it to 10.6 and move the changelog entry there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov merged PR #16119:
URL: https://github.com/apache/lucene/pull/16119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead [lucene]
Posted by "neoremind (via GitHub)" <gi...@apache.org>.
neoremind commented on PR #16119:
URL: https://github.com/apache/lucene/pull/16119#issuecomment-4565701444
Hi @uschindler @msokolov would you mind taking a look at this PR? I noticed you once got involved with this piece of code and relevant PR. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org