You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "mikemccand (via GitHub)" <gi...@apache.org> on 2026/01/07 12:11:11 UTC
[I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
mikemccand opened a new issue, #15552:
URL: https://github.com/apache/lucene/issues/15552
### Description
Lucene uses CRC32 for its end-to-end checksumming. Every index file records its own checksum on write, and when lighting a new segment in `IndexReader` we also validate when we can. `CheckIndex` validates all files. `IndexWriter` validates source segment files when merging.
It's awesome, it catches insidious bit flips for [those people still not using ECC RAM](https://www.reddit.com/r/hardware/comments/y5yroa/linus_tolvards_is_upgrading_his_computer_with_ecc/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button). I know at least @rmuir and myself and maybe @uschindler (?) have been hit by intermittent bad RAM. Once it strikes you personally you will never again tolerate non-ECC RAM on your dev boxes hah.
At Amazon customer facing product search, we validate Lucene's checksum through each step of our near-real-time segment replication (via S3), so we can (hopefully -- there are still technically vulnerabilities if you have adversarial bit-flipping RAM monster lurking in your box (see visuals from [Gemini](https://gemini.google.com/share/0d38531a04ce) and [Grok](https://grok.com/imagine/post/6dfce282-2288-4c8d-a2da-b4d61ecc0a5a?source=copy_link&platform=ios&t=1dce97118e9c))) prevent any bit flips from metastasizing into our persistent S3 index snapshots. This really matters at Amazon's crazy scale (~ 10s of PiB replicated per day to/from S3 -- many chances for errant bit flips!). S3 write/read also does its own checksumming, but that won't catch bit flips on the `IndexWriter` node before a file is uploaded, so we use/validate both checksums.
One of the delightful properties of CRC32 (thank you MATH!) is you can map/reduce it!
I.e. slice up a large (5 GB is Lucene's default max merged segment) segment into N chunks, compute CRC32 of each chunk concurrently, and then use `crc32_combine` zlib API to merge the N CRC32s into a single CRC32 that matches the whole file's sequential CRC32 checksum. It's quite simple (says [Claude](https://claude.ai/share/e2459acc-a812-4a21-9489-680c50a29471) and [Grok](https://grok.com/share/c2hhcmQtMw_610209f9-b4bd-41fe-9e90-b11f5fae9395)). This would be awesome because then users like us could use chunk'd upload/download to improve aggregate S3 throughput and lower the nrt refresh latency during replication.
But, annoyingly, it looks like JDK's CRC32 implementation (`java.util.zip.CRC32`) does not expose `crc32_combine`?
Maybe Lucene could provide a utility class to make this simple? Claude provides a [java implementation of `crc32_combine`](https://claude.ai/share/e2459acc-a812-4a21-9489-680c50a29471), perhaps it is hallucination free? Or maybe we could use FFM to access the `crc32_combine` -- but I'm not sure we can rely on it always being accessible/visible to the JVM, always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3751609215
> Have we ever changed the checksummer? Has it been CRC32 since the start (I think so?).
I really think CRC32C would be better, question is, is it worth the effort required to move over?
Also related to this, at some point we should double-check the math. IIRC these algorithms are only appropriate for file sizes up to some certain limit (e.g. some number of GB). Otherwise they may not detect problems reliably.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3751529121
> For speeding up the current one, maybe look into bumping this one to 2KB: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/BufferedChecksum.java#L32
>
> 1KB value was benchmarked a long time ago with an AVX-128 or AVX-256 machine. These days people have 64-byte vectors.
Oh yeah +1 to explore this ... but I think it's crazy fast today? At least I haven't seen / don't remember it being unexpectedly costly? I'll open a spinoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "Tony-X (via GitHub)" <gi...@apache.org>.
Tony-X commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3720233484
There are some relevant code in the AWS Java SDK v2 which seems to contain efforts to make the combining (reducing) routine faster.
https://github.com/aws/aws-sdk-java-v2/blob/84d5c3453732268187920125c4ab796870618ebb/core/checksums/src/main/java/software/amazon/awssdk/checksums/internal/SdkCrc32Checksum.java#L590
https://github.com/aws/aws-sdk-java-v2/blob/master/core/checksums/src/main/java/software/amazon/awssdk/checksums/internal/CrcCombineChecksumUtil.java#L31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3718733187
Wah this is several loops, not nice code. Sorry ignore my previous comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3751525420
> I'm trying to understand the use-case, for s3 wouldn't you use it's native checksum functionality instead?
Yeah we do also use S3's checksums ... I thought it was still helpful to also validate Lucene's checksum? We confirm checksum each step of a segment file's journey...
Otherwise we'd be vulnerable to bit flips that happen after `IndexWriter` successfully writes the file, but before it is copied out to S3? E.g. if the local storage (where `IndexWriter` writes the index) is a bit-flipper, or the RAM of the indexer box, when we read the bytes and send to S3, if what we sent to S3 has a bit flipped, S3 would think it's fine (its checksum matches what we sent) and we'd have a corrupt index replicated to S3. Downloading would bring the corruption to replicas and S3 checksum is still fine (S3 stores the bit-flipped version).
Lucene might detect the bit flip when a replica lights the segment, if it was in a small/metadata index file (that we check immediately on opening the segment files), but if it's in a large file, it won't be caught right away? Though if replica hits certin exceptions, Lucene will go validate checksum I think? (Or is that only when indexing?)
But ... if the indexing box's RAM or local storage is a bit-flipper, chances are `IndexWriter` would eventually detect the corruption (e.g. we `checkIntegritry` of all segment files when merging)... the chances of a bit flipping in that window before/during S3 replication and NOT being hit by `IndexWriter` is probably low in general. So yeah maybe we should turn off validating Lucene checksums and rely entirely on S3, then we can use S3's chunk'd uploading ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3718708184
Here is the original C code in the ZLib or Mark Adler. I don't think we need to use FFM, it is basically just a one-liner with the code you oosted before:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3720969176
For speeding up the current one, maybe look into bumping this one to 2KB: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/BufferedChecksum.java#L32
1KB value was benchmarked a long time ago with an AVX-128 or AVX-256 machine. These days people have 64-byte vectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3751564845
> There are some relevant code in the AWS Java SDK v2 which seems to contain efforts to make the combining (reducing) routine faster.
Whoa, are these the methods that could combine two CRC32 checksums? It sure looks like it ... but they are marked as `@SdkInternalApi` ... though the comment says it was taken from `zlip`:
```
* The implementation of CRC combination was taken from the zlib source code here:
* <a href="https://github.com/luvit/zlib/blob/master/crc32.c">https://github.com/luvit/zlib/blob/master/crc32.c</a>
```
But:
> But we should be careful about exposing too much here? I feel like lucene should be free to change the polynomial if we want, for improved performance (e.g. switch to java.util.zip.CRC32C). It is hardware-accelerated via dedicated insns on all modern cpus.
Hmm that is a good point -- the choice of checksum algo/polynomial/matrices is really an implementation detail -- if we expose this utility API it would hamper (maybe?) what checksum algos we could use in the future. And one could always do this concurrent checksum chunking outside of Lucene hardwired to CRC32 and accept the risk that Lucene can freely change it at any time ... so maybe we should not pursue this. Have we ever changed the checksummer? Has it been CRC32 since the start (I think so?).
Anyway I'm OK with WONTFIX -- thanks all for the discussion / pointers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3720950820
Yeah i haven't had the time to benchmark the CRC32C openjdk code, I wouldn't be surprised if there is a speedup, so I can't be sure if it is worth the trouble or not. The currently CRC32 is already vectorized and very fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3773216676
> Also related to this, at some point we should double-check the math. IIRC these algorithms are only appropriate for file sizes up to some certain limit (e.g. some number of GB). Otherwise they may not detect problems reliably.
I tried to [ask Claude Opus 4.5 about this](https://claude.ai/share/91bf5b0b-3d76-4fbc-aad4-4893cc0168e8) but its response is confusing :) I think the problem is "false negative" risk meaning bit(s) flipped but the bit-flipped file has the same checksum as the original file ("checksum collision") so the error goes undetected. The risk of this is ~1 in ~4.3 billion (2 ^ 32) assuming your bit flips have no accidental (or adversarial -- this is not a secure/crypto hash?) correlation with CRC32's collisions.
But this is a risk per checksum+validation right, not "per GB of file you are checksumming" or so? So, as long as Lucene isn't using billions of files in an index, the risk remains lowish for any single Lucene user?
So, of the billions of Lucene users ;) (well, individual index files written/read times all Lucene usage integrated over time since we added checksums), some have probably hit this false negative! Oh wait, the universe is smaller -- it's only those segment files written with a bit-flipper in the path?
Still, for such users, it's likely their bit-flipper (wherever it is -- RAM, bus, storage, CPU cache lines) will still be detected even if they unluckily hit the jackpot once (checkpoint collision). Weird/scary/hard to think about...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3720682622
> But we should be careful about exposing too much here? I feel like lucene should be free to change the polynomial if we want, for improved performance (e.g. switch to java.util.zip.CRC32C). It is hardware-accelerated via dedicated insns on all modern cpus.
Yes! We can change or fileformats with some extra version number in the checksum header two switch checkums. The code for combining them is same only the `POLY` constant is different depending on CRC32 variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3719322005
I'm trying to understand the use-case, for s3 wouldn't you use it's native checksum functionality instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Provide helper utility API to compute CRC32 across concurrent (map/reduce) chunks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #15552:
URL: https://github.com/apache/lucene/issues/15552#issuecomment-3719517277
Oh i see, i didnt realize they had pluggable algorithms: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
But we should be careful about exposing too much here? I feel like lucene should be free to change the polynomial if we want, for improved performance (e.g. switch to java.util.zip.CRC32C). It is hardware-accelerated via dedicated insns on all modern cpus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org