You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "houseme (via GitHub)" <gi...@apache.org> on 2026/09/05 06:28:49 UTC
[PR] Improve Snappy CRC32 with crc-fast [avro-rs]
houseme opened a new pull request, #660:
URL: https://github.com/apache/avro-rs/pull/660
## Summary
- replace the optional Snappy `crc32fast` dependency with `crc-fast` 1.10.0
- use `crc_fast::crc32_iso_hdlc(...)` for Avro Snappy block CRC32 calculation
- add a regression test for the standard CRC-32/ISO-HDLC trailer bytes
- bump workspace and CI MSRV from 1.88.0 to 1.89.0 because `crc-fast` 1.10.0 requires Rust 1.89
Closes #659.
## Rationale
The Snappy codec only needs CRC32 over contiguous decoded/uncompressed blocks, so `crc-fast`'s specialized one-shot `crc32_iso_hdlc` helper fits the existing call sites directly. The algorithm remains CRC-32/ISO-HDLC, and the new test locks the public Avro trailer format to the standard `b"123456789" -> 0xcbf43926` check value encoded as big-endian bytes.
For this exact usage pattern, a local microbenchmark on Darwin arm64 (`Mac17,6`, `rustc 1.98.1`) showed `crc-fast` improving throughput over the current `crc32fast::Hasher::new(); update(); finalize()` flow:
| Block size | crc32fast | crc-fast | Speedup |
| ---: | ---: | ---: | ---: |
| 128 B | 12.12 GiB/s | 21.02 GiB/s | 1.73x |
| 1 KiB | 10.93 GiB/s | 70.03 GiB/s | 6.41x |
| 16 KiB | 22.56 GiB/s | 81.98 GiB/s | 3.63x |
| 1 MiB | 33.27 GiB/s | 101.22 GiB/s | 3.04x |
| 16 MiB | 33.42 GiB/s | 97.76 GiB/s | 2.93x |
`crc-fast` is configured with `default-features = false, features = ["std"]` to avoid its default `ffi`/`panic-handler` features while retaining std-backed runtime SIMD detection.
## Testing
- `cargo check -p apache-avro --features snappy`
- `cargo test -p apache-avro --features snappy,derive snappy`
- `cargo fmt --all --check`
- `cargo build --all-features --all-targets`
- `cargo check -p apache-avro --target aarch64-unknown-linux-gnu --features snappy`
Not completed locally:
- `wasm32-unknown-unknown` CI target validation. Installing the target locally stalled while downloading `rust-std`, so this should still be watched in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "houseme (via GitHub)" <gi...@apache.org>.
houseme commented on PR #660:
URL: https://github.com/apache/avro-rs/pull/660#issuecomment-5551450542
Yes. I added a dedicated benchmark for this in `avro/benches/crc32_snappy.rs`.
It now measures three paths on both aarch64 and x86_64:
- raw CRC-32/ISO-HDLC;
- Snappy compression plus the Avro CRC32 trailer;
- Snappy decompression plus CRC32 verification.
The results are documented in `avro/benches/crc32_snappy_results.md`. The short version is that the isolated CRC gain does carry into the Snappy-shaped encode/decode paths, especially from 1 KiB upward. On the tested aarch64 machine, Snappy compress + CRC improved up to 1.58x and decompress + CRC up to 2.29x. On the tested x86_64 EPYC host, compress + CRC improved up to 4.32x and decompress + CRC up to 3.72x.
The SIMD detail is architecture-specific: on aarch64 the improvement comes from `crc-fast` using PMULL/AES folding, with SHA3/EOR3-assisted variants when available. On x86_64, SSE4.2 CRC32 is CRC32C/ISCSI rather than CRC-32/ISO-HDLC, so the relevant acceleration is the PCLMULQDQ / VPCLMULQDQ carry-less multiply family.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g merged PR #660:
URL: https://github.com/apache/avro-rs/pull/660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #660:
URL: https://github.com/apache/avro-rs/pull/660#issuecomment-5681874036
Thank you, @houseme !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #660:
URL: https://github.com/apache/avro-rs/pull/660#issuecomment-5678638043
> If @martin-g agrees with the MSRV bump from `1.88` to `1.89` this can be merged
Sure!
My approach so far was to support `stable` - 5, i.e. current stable is 1.98, so Avro MSRV could be even 1.93.
There is no hard rule here. Just no need to require very recent version if apache_avro or its dependencies do not need something from recent Rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #660:
URL: https://github.com/apache/avro-rs/pull/660#issuecomment-5550951593
Do you actually measure a difference when decoding or encoding Avro files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Snappy CRC32 with crc-fast [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #660:
URL: https://github.com/apache/avro-rs/pull/660#issuecomment-5678645846
@houseme There is a merge conflict in Cargo.lock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org