You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jecsand838 (via GitHub)" <gi...@apache.org> on 2025/08/09 05:54:29 UTC
[PR] Refactor Avro `Decoder` to support partial decoding and improve decod… [arrow-rs]
jecsand838 opened a new pull request, #8100:
URL: https://github.com/apache/arrow-rs/pull/8100
# Which issue does this PR close?
- Part of https://github.com/apache/arrow-rs/issues/4886
# Rationale for this change
Decoding Avro **single-object encoded** streams was brittle when data arrived in partial chunks (e.g., from async or networked sources). The old implementation relied on ad‑hoc prefix handling and assumed a full record would be available, producing hard errors for otherwise normal “incomplete buffer” situations. Additionally, the Avro OCF (Object Container File) path iterated record‑by‑record through a shared row decoder, adding overhead.
This PR introduces a small state machine for single‑object decoding and a block‑aware path for OCF, making streaming more robust and OCF decoding more efficient while preserving the public API surface.
# What changes are included in this PR?
**Single‑object decoding (streaming)**
- Replace ad‑hoc prefix parsing (`expect_prefix`, `handle_prefix`, `handle_fingerprint`) with an explicit state machine:
- New `enum DecoderState { Magic, Fingerprint, Record, SchemaChange, Finished }`.
- `Decoder` now tracks `state`, `bytes_remaining`, and a `fingerprint_buf` to incrementally assemble the fingerprint.
- New helper `is_incomplete_data(&ArrowError) -> bool` to treat “Unexpected EOF”, “bad varint”, and “offset overflow” as *incomplete input* instead of fatal errors.
- Reworked `Decoder::decode(&[u8]) -> Result<usize, ArrowError>`:
- Consumes data according to the state machine.
- Cleanly returns when more bytes are needed (no spurious errors for partial chunks).
- Defers schema switching until after flushing currently decoded rows.
- Updated `Decoder::flush()` to emit a batch only when rows are ready and to transition the state correctly (including a staged `SchemaChange`).
**OCF (Object Container File) decoding**
- Add block‑aware decoding methods on `Decoder` used by `Reader`:
- `decode_block(&[u8], count: usize) -> Result<(consumed, records_decoded), ArrowError>`
- `flush_block() -> Result<Option<RecordBatch>, ArrowError>`
- `Reader` now tracks `block_count` and decodes up to the number of records in the current block, reducing per‑row overhead and improving throughput.
- `ReaderBuilder::build` initializes the new `block_count` path.
**API / struct adjustments**
- Remove internal `expect_prefix` flag from `Decoder`; behavior is driven by the state machine.
- `ReaderBuilder::make_decoder_with_parts` updated accordingly (no behavior change to public builder methods).
- No public API signature changes for `Reader`, `Decoder`, or `ReaderBuilder`.
**Tests**
- Add targeted streaming tests:
- `test_two_messages_same_schema`
- `test_two_messages_schema_switch`
- `test_split_message_across_chunks`
- Update prefix‑handling tests to validate state transitions (`Magic` → `Fingerprint`, etc.) and new error messages.
- Retain and exercise existing suites (types, lists, nested structures, decimals, enums, strict mode) with minimal adjustments.
# Are these changes tested?
Yes.
- New unit tests cover:
- Multi‑message streams with/without schema switches
- Messages split across chunk boundaries
- Incremental prefix/fingerprint parsing
- Existing tests continue to cover OCF reading, compression, complex/nested types, strict mode, etc.
- The new OCF path is exercised by the unchanged OCF tests since `Reader` now uses `decode_block/flush_block`.
# Are there any user-facing changes?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267516616
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -130,6 +129,16 @@ fn read_header<R: BufRead>(mut reader: R) -> Result<Header, ArrowError> {
})
}
+fn is_incomplete_data(err: &ArrowError) -> bool {
+ matches!(
+ err,
+ ArrowError::ParseError(msg)
+ if msg.contains("Unexpected EOF")
+ || msg.contains("bad varint")
+ || msg.contains("offset overflow")
Review Comment:
That's fair. I planned to clean this up in the Schema Resolution `RecordDecoder` PR that's coming after https://github.com/apache/arrow-rs/pull/8047 get merged in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2268135694
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -302,6 +306,11 @@ impl Decoder {
self.remaining_capacity == 0
}
+ /// Returns true if the decoder has not decoded any batches yet.
+ pub fn batch_is_empty(&self) -> bool {
Review Comment:
I didn't see any harm in leaving it `pub`. Figured it maybe useful for someone, plus `batch_is_full` was already `pub` as well so I was trying to keep it consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265411002
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -282,6 +310,31 @@ impl Decoder {
pub fn batch_is_full(&self) -> bool {
self.remaining_capacity == 0
}
+
+ // Decode either the block count of remaining capacity from `data` (an OCF block payload).
+ //
+ // Returns the number of bytes consumed from `data` along with the number of records decoded.
+ fn decode_block(&mut self, data: &[u8], count: usize) -> Result<(usize, usize), ArrowError> {
Review Comment:
That occurred to me as well. I decided to leave it that way because the only caller is `Reader::read` and `decode_block` is not public.
As an aside, I was unsure of the value a public `decode_block` method would offer. This is due to block encodings only existing in Object Container Files, which the `Reader` handles. In the future if there's demand for decoding blocks outside of the `Reader`, then we'd probably want to refactor the code to support `Decoder::decode_block(block: Block, codec: Option<CompressionCodec>) -> Result<DecodeRes, ArrowError>` or something along those lines like you pointed out. I was just concerned this would be a pre-mature optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265417502
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
Review Comment:
@scovich Yeah that is definitely easier to read imo. Went ahead and pushed those changes up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265405842
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
+ }
+ }
+ }
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) if n > 0 => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
Review Comment:
There's always a magic + fingerprint prefix at the start of each single object encoded record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#issuecomment-3176804613
> Code is good now, except I'm worried about robustness in the face of invalid input bytes.
Ty for the fast follow-up review. Let me know what you think of these latest changes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#issuecomment-3172832866
> Overall looks nice. Some possible simplifications, but my main question/concern is about zero-length records causing zero bytes consumed? The code seems to keep going back and forth on whether it's allowed/possible?
Only reason I added that back in was I didn't think zero byte encodings were legal for single object encodings, but it seems I was wrong and they are. So I'll push up a change to account for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265096937
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
Review Comment:
The callees already (re-)check both these conditions...
```suggestion
self.apply_pending_schema_if_batch_empty();
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -282,6 +310,31 @@ impl Decoder {
pub fn batch_is_full(&self) -> bool {
self.remaining_capacity == 0
}
+
+ // Decode either the block count of remaining capacity from `data` (an OCF block payload).
Review Comment:
```suggestion
// Decode either the block count or remaining capacity from `data` (an OCF block payload).
```
?
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
+ }
+ }
+ }
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) if n > 0 => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
Review Comment:
Is there always a fingerprint after each record? Or just a chance to see a fingerprint?
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
Review Comment:
```suggestion
break; // insufficient bytes
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
+ }
+ }
+ }
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) if n > 0 => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
+ }
+ Ok(_) => {
+ return Err(ArrowError::ParseError(
+ "Record decoder consumed 0 bytes".into(),
Review Comment:
I thought zero-byte records were legal, and we're supposed to keep looping until the output batch is full?
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
Review Comment:
... but this whole control flow is a bit confusing. I wonder if it can be made easier to understand?
```rust
while total_consumed < data.len() && self.remaining_capacity > 0 {
if self.awaiting_body {
let result = match self.active_decoder.decode(...) {
Ok(0) => Err(...),
Ok(n) => {
...
self.awaiting_body = false;
continue; // re-check input buffer before attempting to read the row
}
Err(ref e) if is_incomplete_data(e) => break,
err => err,
}
return result;
}
match self.handle_prefix(...)? {
Some(0) => break, // insufficient bytes
Some(n) => {
total_consumed += n;
self.apply_pending_schema_if_batch_empty();
self.awaiting_body = true;
}
}
}
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -282,6 +310,31 @@ impl Decoder {
pub fn batch_is_full(&self) -> bool {
self.remaining_capacity == 0
}
+
+ // Decode either the block count of remaining capacity from `data` (an OCF block payload).
+ //
+ // Returns the number of bytes consumed from `data` along with the number of records decoded.
+ fn decode_block(&mut self, data: &[u8], count: usize) -> Result<(usize, usize), ArrowError> {
Review Comment:
Two same-type return values with very different meanings... is it worth defining a struct for it so they have names? Or are there few enough (and always internal) callers to keep track of it?
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
Review Comment:
```suggestion
break; // pending schema change ends the batch
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
+ }
+ }
+ }
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) if n > 0 => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
+ }
+ Ok(_) => {
+ return Err(ArrowError::ParseError(
+ "Record decoder consumed 0 bytes".into(),
+ ));
+ }
+ Err(e) => {
+ return if is_incomplete_data(&e) {
+ Ok(total_consumed)
+ } else {
+ Err(e)
+ }
Review Comment:
nit: if/else inside match arms is often bulky and hard to read:
```suggestion
Err(ref e) if is_incomplete_data(&e) => return Ok(total_consumed),
err => return err,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#issuecomment-3180295829
Thanks for the review and code @jecsand838 and @scovich -- let's keep the train moving
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265406153
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -162,29 +171,42 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
- if self.expect_prefix
- && data.len() >= SINGLE_OBJECT_MAGIC.len()
- && !data.starts_with(&SINGLE_OBJECT_MAGIC)
- {
- return Err(ArrowError::ParseError(
- "Expected single‑object encoding fingerprint prefix for first message \
- (writer_schema_store is set but active_fingerprint is None)"
- .into(),
- ));
- }
let mut total_consumed = 0usize;
- // The loop stops when the batch is full, a schema change is staged,
- // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- // We either consumed a prefix (n > 0) and need a schema switch, or we need
- // more bytes to make a decision. Either way, this decoding attempt is finished.
- total_consumed += n;
+ if !self.awaiting_body {
+ if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+ if n == 0 {
+ break;
+ }
+ total_consumed += n;
+ self.awaiting_body = true;
+ if self.remaining_capacity == self.batch_size && self.pending_schema.is_some() {
+ self.apply_pending_schema_if_batch_empty();
+ }
+ if self.remaining_capacity == 0 {
+ break;
+ }
+ }
+ }
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) if n > 0 => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
+ }
+ Ok(_) => {
+ return Err(ArrowError::ParseError(
+ "Record decoder consumed 0 bytes".into(),
Review Comment:
I didn't think they were legal for single object encodings, but sure enough they are. So I'll remove this. Should have re-read the specs lol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#issuecomment-3170390347
@scovich @alamb Here's the follow-up PR with the partial `Decoder` enhancements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#issuecomment-3172856555
@scovich I went ahead and pushed up changes that remove the zero byte error on single object encodings and improve the `Decoder::decode` method's readability.
Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267920792
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
Review Comment:
My latest push includes the `batch_is_empty` helper. That was a good callout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267513571
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
+ }
+
+ /// Produce a `RecordBatch` if at least one row is fully decoded, returning
+ /// `Ok(None)` if no new rows are available.
+ pub fn flush(&mut self) -> Result<Option<RecordBatch>, ArrowError> {
+ if self.remaining_capacity == self.batch_size {
+ return Ok(None);
+ }
+ let batch = self.active_decoder.flush()?;
+ self.remaining_capacity = self.batch_size;
+ self.apply_pending_schema();
Review Comment:
That's a good call out. I'll create a helper for:
```rust
let batch = self.active_decoder.flush()?;
self.remaining_capacity = self.batch_size;
```
I'm wanting to keep the schema change logic completely decoupled from the block decoder/flush path for now. Just to avoid confusion for future contributors and to setup us up for any future `Decoder` decomposition efforts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267919050
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
+ }
+
+ /// Produce a `RecordBatch` if at least one row is fully decoded, returning
+ /// `Ok(None)` if no new rows are available.
+ pub fn flush(&mut self) -> Result<Option<RecordBatch>, ArrowError> {
+ if self.remaining_capacity == self.batch_size {
+ return Ok(None);
+ }
+ let batch = self.active_decoder.flush()?;
+ self.remaining_capacity = self.batch_size;
+ self.apply_pending_schema();
Review Comment:
I just pushed up changes which include a new `flush_and_reset` method:
```rust
fn flush_and_reset(&mut self) -> Result<Option<RecordBatch>, ArrowError> {
if self.batch_is_empty() {
return Ok(None);
}
let batch = self.active_decoder.flush()?;
self.remaining_capacity = self.batch_size;
Ok(Some(batch))
}
```
It abstracted out most of the flush logic and `flush_block` is now just a public wrapper for that new method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2268138211
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -302,6 +306,11 @@ impl Decoder {
self.remaining_capacity == 0
}
+ /// Returns true if the decoder has not decoded any batches yet.
+ pub fn batch_is_empty(&self) -> bool {
Review Comment:
Ah, I didn't realize we already had a `pub batch_is_full` method. Makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2265406406
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -282,6 +310,31 @@ impl Decoder {
pub fn batch_is_full(&self) -> bool {
self.remaining_capacity == 0
}
+
+ // Decode either the block count of remaining capacity from `data` (an OCF block payload).
Review Comment:
Typo on my part. Ty for the catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2266533676
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -173,36 +173,29 @@ impl Decoder {
pub fn decode(&mut self, data: &[u8]) -> Result<usize, ArrowError> {
let mut total_consumed = 0usize;
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if !self.awaiting_body {
- if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
- if n == 0 {
- break;
- }
- total_consumed += n;
- self.awaiting_body = true;
- self.apply_pending_schema_if_batch_empty();
- if self.remaining_capacity == 0 {
- break;
+ if self.awaiting_body {
+ match self.active_decoder.decode(&data[total_consumed..], 1) {
+ Ok(n) => {
+ self.remaining_capacity -= 1;
+ total_consumed += n;
+ self.awaiting_body = false;
+ continue;
}
- }
+ Err(ref e) if is_incomplete_data(e) => return Ok(total_consumed),
Review Comment:
```suggestion
Err(ref e) if is_incomplete_data(e) => break,
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -130,6 +129,16 @@ fn read_header<R: BufRead>(mut reader: R) -> Result<Header, ArrowError> {
})
}
+fn is_incomplete_data(err: &ArrowError) -> bool {
+ matches!(
+ err,
+ ArrowError::ParseError(msg)
+ if msg.contains("Unexpected EOF")
+ || msg.contains("bad varint")
+ || msg.contains("offset overflow")
Review Comment:
Hmm, after thinking more about this over the weekend --
Trying to interpret/suppress these errors will almost certainly make the decoder brittle in the face of malformed input bytes that legitimately trigger these errors. For example, we could put the decoder in an infinite loop where it keeps trying to fetch more and more bytes in hopes of eliminating the error, when the error is fully contained in the existing buffer.
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
+ }
+
+ /// Produce a `RecordBatch` if at least one row is fully decoded, returning
+ /// `Ok(None)` if no new rows are available.
+ pub fn flush(&mut self) -> Result<Option<RecordBatch>, ArrowError> {
+ if self.remaining_capacity == self.batch_size {
+ return Ok(None);
+ }
+ let batch = self.active_decoder.flush()?;
+ self.remaining_capacity = self.batch_size;
+ self.apply_pending_schema();
Review Comment:
`flush` and `flush_block` are identical except this call to `self.apply_pending_schema`?
Is there a way to deduplicate the code? Maybe a `flush_internal` that takes a boolean argument (which the compiler would aggressively inline away as if it were a generic parameter)?
Or just call `self.apply_pending_schema` unconditionally, knowing it should be a no-op during block decoding because `self.pending_schema` is always None?
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
Review Comment:
Also, we have quite a few places that could benefit from a small helper method:
```rust
fn batch_is_empty(&self) -> bool {
self.remaining_capacity == self.batch_size
}
```
... which could improve readability, e.g.
```suggestion
if self.batch_is_empty() {
self.apply_pending_schema();
}
```
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -270,6 +271,24 @@ impl Decoder {
self.active_decoder = new_decoder;
}
}
+ }
+
+ fn apply_pending_schema_if_batch_empty(&mut self) {
+ if self.remaining_capacity != self.batch_size {
+ return;
+ }
+ self.apply_pending_schema();
Review Comment:
```suggestion
if self.remaining_capacity == self.batch_size {
self.apply_pending_schema();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267954606
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -130,6 +129,16 @@ fn read_header<R: BufRead>(mut reader: R) -> Result<Header, ArrowError> {
})
}
+fn is_incomplete_data(err: &ArrowError) -> bool {
+ matches!(
+ err,
+ ArrowError::ParseError(msg)
+ if msg.contains("Unexpected EOF")
+ || msg.contains("bad varint")
+ || msg.contains("offset overflow")
Review Comment:
I ended up modifying `is_incomplete_data` to this:
```
fn is_incomplete_data(err: &ArrowError) -> bool {
matches!(
err,
ArrowError::ParseError(msg)
if msg.contains("Unexpected EOF")
)
}
```
I double checked the `arrow-avro/src/reader/cursor.rs` file and we should only get the `Unexpected EOF` error if there's too few bytes.
I'll also improve the logic in `arrow-avro/src/reader/cursor.rs` to support a more deliberate and less rigid implementation in a future PR. I left a comment in the code calling this out.
Let me know what you think of this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2267516616
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -130,6 +129,16 @@ fn read_header<R: BufRead>(mut reader: R) -> Result<Header, ArrowError> {
})
}
+fn is_incomplete_data(err: &ArrowError) -> bool {
+ matches!(
+ err,
+ ArrowError::ParseError(msg)
+ if msg.contains("Unexpected EOF")
+ || msg.contains("bad varint")
+ || msg.contains("offset overflow")
Review Comment:
That's fair. I planned to clean this up in the Schema Resolution `RecordDecoder` PR that's coming after https://github.com/apache/arrow-rs/pull/8047 get merged in.
However I'll take this out due to the risk of the infinite loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor arrow-avro `Decoder` to support partial decoding [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8100:
URL: https://github.com/apache/arrow-rs/pull/8100#discussion_r2268126358
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -302,6 +306,11 @@ impl Decoder {
self.remaining_capacity == 0
}
+ /// Returns true if the decoder has not decoded any batches yet.
+ pub fn batch_is_empty(&self) -> bool {
Review Comment:
nit: Not sure it needs to be `pub`? But I guess it's plenty well-defined (`batch_is_empty` is true if `flush` would return None), so no harm leaving it public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org