You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/01/22 17:32:42 UTC
[I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
alamb opened a new issue, #9247:
URL: https://github.com/apache/arrow-rs/issues/9247
**Describe the bug**
There is some issue with decoding fixedsizebinary I found while working on tests for other
**To Reproduce**
Add this test to the parquet tests:
```diff
--- a/parquet/src/arrow/arrow_writer/mod.rs
+++ b/parquet/src/arrow/arrow_writer/mod.rs
@@ -2161,8 +2161,11 @@ mod tests {
)
.unwrap();
- let data = DictionaryArray::<K>::new(keys, Arc::new(values));
- let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(data)]).unwrap();
+
+ let data: ArrayRef = Arc::new(DictionaryArray::<K>::new(keys, Arc::new(values)));
+ one_column_roundtrip(Arc::clone(&data), true);
+
+ let batch = RecordBatch::try_new(Arc::new(schema), vec![data]).unwrap();
roundtrip(batch, None);
}
```
Fails like this
```
---- arrow::arrow_writer::tests::test_fixed_size_binary_in_dict stdout ----
thread 'arrow::arrow_writer::tests::test_fixed_size_binary_in_dict' (467576) panicked at parquet/src/arrow/buffer/offset_buffer.rs:133:48:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2")
```
I am pretty sure this is related to disabling dictionary pages
**Expected behavior**
Test should pass
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-3785755314
I found this while working on tests for
- https://github.com/apache/arrow-rs/pull/9246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-3791286567
Thanks @albertlockett 🙏 -- there is no special urgency from me, I just wanted to try and capture this issue that happened to find while writing tests for something different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-4040864200
I would personally start with adding the regression test and then fix the bug. Otherwise this is good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "albertlockett (via GitHub)" <gi...@apache.org>.
albertlockett commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-3790506265
I'd be happy to look into this @alamb. I imagine I'll have some time next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-4018960578
Hi @alamb, I’d like to work on this issue.
From looking into the failure, it seems the problem happens during the Parquet roundtrip when rebuilding a `DictionaryArray` whose values are a `FixedSizeBinaryArray`. The error suggests that the resulting `FixedSizeBinaryArray` ends up with two buffers, even though `FixedSizeBinary` should only have a single data buffer.
My current guess is that somewhere in the Parquet → Arrow decoding path, `FixedSizeBinary` is being handled similarly to variable-length binary types (`Binary` or `LargeBinary`), which expect an offset buffer. However, since `FixedSizeBinary` has a fixed width, it should only use the data buffer.
My plan is to:
1. Trace how the array is reconstructed during the Parquet roundtrip when dictionary encoding is used.
2. Locate where the `ArrayData` for `FixedSizeBinary` is created and check how the buffers are assigned.
3. Make sure `FixedSizeBinary` arrays are built with only the data buffer (no offset buffer).
4. Add a regression test using a `DictionaryArray` with `FixedSizeBinary` values to confirm the roundtrip works correctly.
Does this approach sound reasonable, or is there a specific part of the codebase you’d recommend focusing on? Happy to adjust based on feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-3785727830
The test was originally added by @albertlockett in this PR
- #7446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-4054781339
hi @alamb Thanks for the suggestion!
That makes sense — I'll start by adding a regression test that reproduces the failure and then work on fixing the bug once the test is in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] InvalidArgumentError("Expected 1 buffers in array of type FixedSizeBinary(4), got 2" [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #9247:
URL: https://github.com/apache/arrow-rs/issues/9247#issuecomment-6231348759
- this might be resolved by https://github.com/apache/arrow-rs/pull/11262, to close this it would be good to get confirmation via adding a test (or at least pointing to an existing test)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org