You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jecsand838 (via GitHub)" <gi...@apache.org> on 2025/10/08 22:47:27 UTC
[PR] Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro [arrow-rs]
jecsand838 opened a new pull request, #8575:
URL: https://github.com/apache/arrow-rs/pull/8575
# Which issue does this PR close?
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
- Part of #4886
# Rationale for this change
Many Kafka deployments use **Confluent Schema Registry** (4‑byte, big‑endian ID) and/or **Apicurio Registry** (commonly 8‑byte, big‑endian *global* ID). `arrow-avro` already supported the Confluent wire format; this PR adds first‑class support for Apicurio’s 64‑bit ID, enabling seamless decode/encode of streams that carry an 8‑byte ID after the magic byte.
This improves interoperability with ecosystems that standardize on Apicurio or Red Hat Event Streams.
# What changes are included in this PR?
* **New 64‑bit ID support**
* Add `Fingerprint::Id64(u64)` and `FingerprintAlgorithm::Id64`.
* Add `FingerprintStrategy::Id64(u64)` and helper `Fingerprint::load_fingerprint_id64`.
* Extend `Fingerprint::serialized_prefix` to emit/read an 8‑byte big‑endian ID after the `0x00` magic byte.
* **Clarify/algin numeric‑ID algorithm names**
* Replace the prior `FingerprintAlgorithm::None` (numeric ID) with `FingerprintAlgorithm::Id` (4‑byte) and introduce `Id64` (8‑byte). All examples and call sites updated accordingly (i.e., `SchemaStore::new_with_type(FingerprintAlgorithm::Id)`).
* **Reader/Writer plumbing**
* `Decoder` now understands both `Id` (4‑byte) and `Id64` (8‑byte) prefixes.
* `WriterBuilder` accepts `FingerprintStrategy::Id64` to write frames with a 64‑bit ID.
* **SchemaStore behavior**
* `SchemaStore::register` now errors for `Id`/`Id64` algorithms (as those IDs come from a registry); callers should use `set(Fingerprint::Id(_)|Id64(_), ...)` to associate schemas by registry ID.
* **Docs & examples**
* Reader docs expanded to call out Confluent (4‑byte) and Apicurio (8‑byte) formats; examples switched to `FingerprintAlgorithm::Id`. Bench and example updates reflect the new variants.
# Are these changes tested?
Yes. This PR adds/updates unit tests that exercise the new path end‑to‑end, including:
* `test_stream_writer_with_id64_fingerprint_rt` (writer round‑trip with 64‑bit ID).
* `test_two_messages_same_schema_id64` (decoder round‑trip with 64‑bit ID).
* Adjustments to existing tests and benches to use `FingerprintAlgorithm::Id` instead of `None`.
# Are there any user-facing changes?
N/A because `arrow-avro` isn't public yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro [arrow-rs]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on PR #8575:
URL: https://github.com/apache/arrow-rs/pull/8575#issuecomment-3388649593
Thanks @jecsand838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro [arrow-rs]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #8575:
URL: https://github.com/apache/arrow-rs/pull/8575#discussion_r2416256876
##########
arrow-avro/src/schema.rs:
##########
@@ -539,8 +543,10 @@ pub enum FingerprintAlgorithm {
/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
#[default]
Rabin,
- /// Represents a fingerprint not based on a hash algorithm, (e.g., a 32-bit Schema Registry ID.)
- None,
+ /// Represents a 32 bit fingerprint not based on a hash algorithm, (e.g., a 32-bit Schema Registry ID.)
+ Id,
+ /// Represents a 64 bit fingerprint not based on a hash algorithm, (e.g., a 32-bit Schema Registry ID.)
Review Comment:
```suggestion
/// Represents a 64 bit fingerprint not based on a hash algorithm, (e.g., a 64-bit Schema Registry ID.)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #8575:
URL: https://github.com/apache/arrow-rs/pull/8575#issuecomment-3383477183
@alamb @mbrobbel Do you think we'd be able to get this into v57.0.0?
It's a small diff, but relatively high impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro [arrow-rs]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel merged PR #8575:
URL: https://github.com/apache/arrow-rs/pull/8575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org