You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hzuo (via GitHub)" <gi...@apache.org> on 2024/03/26 00:04:20 UTC
[PR] fix [arrow-rs]
hzuo opened a new pull request, #5554:
URL: https://github.com/apache/arrow-rs/pull/5554
# Which issue does this PR close?
Closes #5553
# Rationale for this change
`FileWriter` and `StreamWriter` should ensure that the data is written with appropriately alignment such that arrays can be used without copying to a more-aligned buffer.
In particular, as of Rust 1.77.0 and LLVM 18, `i128` now has a 16-byte alignment requirement, i.e. `std::mem::align_of::<i128> == 16`. So `Decimal128Array`s must be aligned to a 16-byte boundary when it serialized into an IPC buffer. The `pad_to_8` used everywhere in the IPC code causes it to pad insufficiently.
(copied from issue)
# What changes are included in this PR?
- Make all the IPC code actually respect the `alignment` specified in `IpcWriteOptions`
- Add new `enforce_zero_copy` option to `FileDecoder`, `read_record_batch`, and `read_dictionary`, which enforces that the read is a zero-copy read. Previously if the buffer was unaligned then the code does a memcpy to an aligned buffer, which users of Arrow as a zero-copy format may not expect / may want to enforce does not happen.
# Are there any user-facing changes?
Yes:
- `alignment` in `IpcWriteOptions::try_with` is now a u8 inside of a usize (breaking but minor)
- New `enforce_zero_copy` argument on `read_record_batch` and `read_dictionary` (breaking but minor)
- New option `with_enforce_zero_copy` to FileDecoder (non-breaking)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546785723
##########
arrow-ipc/src/reader.rs:
##########
@@ -628,6 +688,19 @@ impl FileDecoder {
self
}
+ /// Specify whether to enforce zero copy, which means the array data inside the record batches
+ /// produced by this decoder will always reference the input buffer.
Review Comment:
Yeah I agree, will make this change.
My original thinking was that there might be a variety of reasons why IPC decoders may need to create a copy of the input data, and this would be a catch-all so that the users don't have to understand the intricacies of why or why not copies need to be made.
But agree that it ends up being kinda vague, and in the future if there are other reasons copies need to be made, probably better to expose each of the "reasons" as a granular "requirement" - if there ends up being a ton, we can expose a catch-all, but no reason to anticipate that today at the expense of clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1540157385
##########
arrow-buffer/src/buffer/scalar.rs:
##########
@@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
- /// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
+ /// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
+ assert_eq!(
+ buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
+ 0,
+ "buffer is not aligned"
+ );
Review Comment:
but a user could call `new` without going through the From conversion, right? and the docs are specifically talking about `new`?
also looks like `From<Buffer> for ScalarBuffer<T>` doesn't go through `new` so this wouldn't add any additional overhead
but not opinionated about this at all - happy to remove if you'd like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment when reading IPC, respect alignment when writing [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1551761295
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
Filed https://github.com/apache/arrow-rs/issues/5588 to track a potentially new API. Thanks @hzuo and @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1549239776
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
This is a breaking change to a public method, I think it would be good to find a way to avoid this. Perhaps by encapsulating some or all of the arguments onto a struct or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1538444049
##########
arrow-ipc/src/writer.rs:
##########
@@ -1406,17 +1437,20 @@ fn write_buffer(
// make new index entry
buffers.push(crate::Buffer::new(offset, len));
- // padding and make offset 8 bytes aligned
- let pad_len = pad_to_8(len as u32) as i64;
- arrow_data.extend_from_slice(&vec![0u8; pad_len as usize][..]);
+ // padding and make offset aligned
+ let pad_len = pad_to_alignment(alignment, len as usize);
+ arrow_data.extend_from_slice(&PADDING[..pad_len]);
- Ok(offset + len + pad_len)
+ Ok(offset + len + (pad_len as i64))
}
-/// Calculate an 8-byte boundary and return the number of bytes needed to pad to 8 bytes
+const PADDING: [u8; 64] = [0; 64];
Review Comment:
previously a Vec was being allocated every time padding was needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1540173725
##########
arrow-buffer/src/buffer/scalar.rs:
##########
@@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
- /// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
+ /// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
+ assert_eq!(
+ buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
+ 0,
+ "buffer is not aligned"
+ );
Review Comment:
The `.into()` call a few lines below invokes the From conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546746985
##########
arrow-ipc/src/reader.rs:
##########
@@ -628,6 +688,19 @@ impl FileDecoder {
self
}
+ /// Specify whether to enforce zero copy, which means the array data inside the record batches
+ /// produced by this decoder will always reference the input buffer.
Review Comment:
What do you think about calling this concept / field `require_alignment` which describes what is required (rather than how it is enforced)?
It might also help make the intent clearer to users (that the IPC decoder will error if the buffers are not aligned properly)
I don't feel super strongly about this, but I was a little confused about what `enforce_zero_copy` meant until I found this comment
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
)
.unwrap();
assert_eq!(batch, roundtrip);
}
+ #[test]
Review Comment:
❤️
##########
arrow-ipc/src/reader.rs:
##########
@@ -396,6 +451,7 @@ pub fn read_record_batch(
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ enforce_zero_copy: bool,
Review Comment:
Since `read_record_batch` is `pub` I think we should also document the parameter here as well (maybe link back to the array builder's docs)
https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/fn.read_record_batch.html
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
Review Comment:
does this fail if you pass `true`?
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
)
.unwrap();
assert_eq!(batch, roundtrip);
}
+ #[test]
+ fn test_unaligned_throws_error_with_enforce_zero_copy() {
+ let batch = RecordBatch::try_from_iter(vec![(
+ "i32",
+ Arc::new(Int32Array::from(vec![1, 2, 3, 4])) as _,
+ )])
+ .unwrap();
+
+ let gen = IpcDataGenerator {};
+ let mut dict_tracker = DictionaryTracker::new(false);
+ let (_, encoded) = gen
+ .encoded_batch(&batch, &mut dict_tracker, &Default::default())
+ .unwrap();
+
+ let message = root_as_message(&encoded.ipc_message).unwrap();
+
+ // Construct an unaligned buffer
+ let mut buffer = MutableBuffer::with_capacity(encoded.arrow_data.len() + 1);
+ buffer.push(0_u8);
+ buffer.extend_from_slice(&encoded.arrow_data);
+ let b = Buffer::from(buffer).slice(1);
+ assert_ne!(b.as_ptr().align_offset(8), 0);
+
+ let ipc_batch = message.header_as_record_batch().unwrap();
+ let result = read_record_batch(
+ &b,
+ ipc_batch,
+ batch.schema(),
+ &Default::default(),
+ None,
+ &message.version(),
+ true,
+ );
+
+ let error = result.unwrap_err();
+ match error {
+ ArrowError::InvalidArgumentError(e) => {
+ assert!(e.contains("Misaligned"));
+ assert!(e.contains("offset from expected alignment of"));
+ }
+ _ => panic!("Expected InvalidArgumentError"),
+ }
Review Comment:
Another way to test this is to do something like
```rust
let err = result.unwrap_err();
assert_eq!(err.to_string(), "expected error string")
```
This also works I just figured I would pass it along
##########
arrow-ipc/src/writer.rs:
##########
@@ -2234,4 +2270,128 @@ mod tests {
let in_batch = RecordBatch::try_new(schema, vec![values]).unwrap();
roundtrip_ensure_sliced_smaller(in_batch, 1000);
}
+
+ #[test]
+ fn test_decimal128_alignment16() {
+ const IPC_ALIGNMENT: u8 = 16;
+
+ // Test a bunch of different dimensions to ensure alignment is never an issue.
+ // For example, if we only test `num_cols = 1` then even with alignment 8 this
+ // test would _happen_ to pass, even though for different dimensions like
+ // `num_cols = 2` it would fail.
+ for num_cols in 1..100 {
+ let num_rows = (num_cols * 7 + 11) % 100; // Deterministic swizzle
+
+ let mut fields = Vec::new();
+ let mut arrays = Vec::new();
+ for i in 0..num_cols {
+ let field = Field::new(&format!("col_{}", i), DataType::Decimal128(38, 10), true);
+ let array = Decimal128Array::from(vec![num_cols as i128; num_rows]);
+ fields.push(field);
+ arrays.push(Arc::new(array) as Arc<dyn Array>);
+ }
+ let schema = Schema::new(fields);
+ let batch = RecordBatch::try_new(Arc::new(schema), arrays).unwrap();
+
+ let mut writer = FileWriter::try_new_with_options(
+ Vec::new(),
+ batch.schema_ref(),
+ IpcWriteOptions::try_new(IPC_ALIGNMENT, false, MetadataVersion::V5).unwrap(),
+ )
+ .unwrap();
+ writer.write(&batch).unwrap();
+ writer.finish().unwrap();
+
+ let out: Vec<u8> = writer.into_inner().unwrap();
+
+ let buffer = Buffer::from_vec(out);
+ let trailer_start = buffer.len() - 10;
+ let footer_len =
+ read_footer_length(buffer[trailer_start..].try_into().unwrap()).unwrap();
+ let footer =
+ root_as_footer(&buffer[trailer_start - footer_len..trailer_start]).unwrap();
+
+ let schema = fb_to_schema(footer.schema().unwrap());
+ assert_eq!(&schema, batch.schema().as_ref());
+
+ let decoder =
Review Comment:
Maybe worth a comment here that the test is that the validation is happening here
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
)
.unwrap();
assert_eq!(batch, roundtrip);
}
+ #[test]
+ fn test_unaligned_throws_error_with_enforce_zero_copy() {
+ let batch = RecordBatch::try_from_iter(vec![(
+ "i32",
+ Arc::new(Int32Array::from(vec![1, 2, 3, 4])) as _,
+ )])
+ .unwrap();
+
+ let gen = IpcDataGenerator {};
+ let mut dict_tracker = DictionaryTracker::new(false);
+ let (_, encoded) = gen
+ .encoded_batch(&batch, &mut dict_tracker, &Default::default())
+ .unwrap();
+
+ let message = root_as_message(&encoded.ipc_message).unwrap();
+
+ // Construct an unaligned buffer
+ let mut buffer = MutableBuffer::with_capacity(encoded.arrow_data.len() + 1);
+ buffer.push(0_u8);
Review Comment:
👍
##########
arrow-ipc/src/writer.rs:
##########
@@ -83,13 +84,13 @@ impl IpcWriteOptions {
}
/// Try create IpcWriteOptions, checking for incompatible settings
pub fn try_new(
- alignment: usize,
+ alignment: u8,
Review Comment:
this is a breaking API change. Since it is fallable anyways, could you perhaps leave the argument a `usize` and throw an error if it is out of range (aka `let alignment = u8::try_from(alignment).map_err(|e| ...)`?)
##########
arrow-integration-testing/src/flight_server_scenarios/integration_test.rs:
##########
@@ -308,6 +308,7 @@ async fn record_batch_from_message(
dictionaries_by_id,
None,
&message.version(),
+ false,
Review Comment:
if these are true, do the tests fail?
##########
arrow-ipc/src/writer.rs:
##########
@@ -2234,4 +2270,128 @@ mod tests {
let in_batch = RecordBatch::try_new(schema, vec![values]).unwrap();
roundtrip_ensure_sliced_smaller(in_batch, 1000);
}
+
+ #[test]
+ fn test_decimal128_alignment16() {
+ const IPC_ALIGNMENT: u8 = 16;
+
+ // Test a bunch of different dimensions to ensure alignment is never an issue.
+ // For example, if we only test `num_cols = 1` then even with alignment 8 this
+ // test would _happen_ to pass, even though for different dimensions like
+ // `num_cols = 2` it would fail.
+ for num_cols in 1..100 {
Review Comment:
do we really need to test 100 columns? That seems overly compute sensitive for relatively small additional coverage. Maybe we could use a few few distinct ones like `1, 17, 23, 73` ?
##########
arrow-ipc/src/reader.rs:
##########
@@ -456,6 +512,7 @@ pub fn read_dictionary(
schema: &Schema,
dictionaries_by_id: &mut HashMap<i64, ArrayRef>,
metadata: &MetadataVersion,
+ enforce_zero_copy: bool,
Review Comment:
likewise this is a public function as well so I think the parameter should be documented
##########
arrow-ipc/src/reader.rs:
##########
@@ -628,6 +688,19 @@ impl FileDecoder {
self
}
+ /// Specify whether to enforce zero copy, which means the array data inside the record batches
+ /// produced by this decoder will always reference the input buffer.
+ ///
+ /// In particular, this means that if there is data that is not aligned properly in the
+ /// input buffer, the decoder will throw rather than copy the data to an aligned buffer.
Review Comment:
```suggestion
/// input buffer, the decoder will return an error rather than copy the data to an aligned buffer.
```
##########
arrow-ipc/src/writer.rs:
##########
@@ -2234,4 +2270,128 @@ mod tests {
let in_batch = RecordBatch::try_new(schema, vec![values]).unwrap();
roundtrip_ensure_sliced_smaller(in_batch, 1000);
}
+
+ #[test]
+ fn test_decimal128_alignment16() {
+ const IPC_ALIGNMENT: u8 = 16;
+
+ // Test a bunch of different dimensions to ensure alignment is never an issue.
+ // For example, if we only test `num_cols = 1` then even with alignment 8 this
+ // test would _happen_ to pass, even though for different dimensions like
+ // `num_cols = 2` it would fail.
+ for num_cols in 1..100 {
+ let num_rows = (num_cols * 7 + 11) % 100; // Deterministic swizzle
+
+ let mut fields = Vec::new();
+ let mut arrays = Vec::new();
+ for i in 0..num_cols {
+ let field = Field::new(&format!("col_{}", i), DataType::Decimal128(38, 10), true);
+ let array = Decimal128Array::from(vec![num_cols as i128; num_rows]);
+ fields.push(field);
+ arrays.push(Arc::new(array) as Arc<dyn Array>);
+ }
+ let schema = Schema::new(fields);
+ let batch = RecordBatch::try_new(Arc::new(schema), arrays).unwrap();
+
+ let mut writer = FileWriter::try_new_with_options(
+ Vec::new(),
+ batch.schema_ref(),
+ IpcWriteOptions::try_new(IPC_ALIGNMENT, false, MetadataVersion::V5).unwrap(),
+ )
+ .unwrap();
+ writer.write(&batch).unwrap();
+ writer.finish().unwrap();
+
+ let out: Vec<u8> = writer.into_inner().unwrap();
+
+ let buffer = Buffer::from_vec(out);
+ let trailer_start = buffer.len() - 10;
+ let footer_len =
+ read_footer_length(buffer[trailer_start..].try_into().unwrap()).unwrap();
+ let footer =
+ root_as_footer(&buffer[trailer_start - footer_len..trailer_start]).unwrap();
+
+ let schema = fb_to_schema(footer.schema().unwrap());
+ assert_eq!(&schema, batch.schema().as_ref());
Review Comment:
I am not sure how much value these checks add -- they are redundant with other code I think and somewhat obscure the point of the test (to set with_enforce_zero_copy and read the data back without error)
##########
arrow-ipc/src/reader.rs:
##########
@@ -71,7 +71,11 @@ fn read_buffer(
/// - check if the bit width of non-64-bit numbers is 64, and
/// - read the buffer as 64-bit (signed integer or float), and
/// - cast the 64-bit array to the appropriate data type
-fn create_array(reader: &mut ArrayReader, field: &Field) -> Result<ArrayRef, ArrowError> {
+fn create_array(
Review Comment:
Can you please add some documentation on what `enforce_zero_copy` does to the docstrings?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1538388341
##########
arrow-ipc/src/writer.rs:
##########
@@ -2234,4 +2270,124 @@ mod tests {
let in_batch = RecordBatch::try_new(schema, vec![values]).unwrap();
roundtrip_ensure_sliced_smaller(in_batch, 1000);
}
+
+ #[test]
+ fn test_decimal128_alignment16() {
+ const IPC_ALIGNMENT: u8 = 16;
+
+ for num_cols in 1..100 {
+ let num_rows = (num_cols * 7 + 11) % 100; // Deterministic swizzle
+
+ let mut fields = Vec::new();
+ let mut arrays = Vec::new();
+ for i in 0..num_cols {
+ let field = Field::new(&format!("col_{}", i), DataType::Decimal128(38, 10), true);
+ let array = Decimal128Array::from(vec![num_cols as i128; num_rows]);
+ fields.push(field);
+ arrays.push(Arc::new(array) as Arc<dyn Array>);
+ }
+ let schema = Schema::new(fields);
+ let batch = RecordBatch::try_new(Arc::new(schema), arrays).unwrap();
+
+ let mut writer = FileWriter::try_new_with_options(
+ Vec::new(),
+ batch.schema_ref(),
+ IpcWriteOptions::try_new(IPC_ALIGNMENT, false, MetadataVersion::V5).unwrap(),
+ )
+ .unwrap();
+ writer.write(&batch).unwrap();
+ writer.finish().unwrap();
+
+ let out: Vec<u8> = writer.into_inner().unwrap();
+
+ let buffer = Buffer::from_vec(out);
+ let trailer_start = buffer.len() - 10;
+ let footer_len =
+ read_footer_length(buffer[trailer_start..].try_into().unwrap()).unwrap();
+ let footer =
+ root_as_footer(&buffer[trailer_start - footer_len..trailer_start]).unwrap();
+
+ let schema = fb_to_schema(footer.schema().unwrap());
+ assert_eq!(&schema, batch.schema().as_ref());
+
+ let decoder =
+ FileDecoder::new(Arc::new(schema), footer.version()).with_enforce_zero_copy(true);
+
+ assert_eq!(footer.dictionaries().unwrap().len(), 0);
+
+ let batches = footer.recordBatches().unwrap();
+ assert_eq!(batches.len(), 1);
+
+ let block = batches.get(0);
+ let block_len = block.bodyLength() as usize + block.metaDataLength() as usize;
+ let data = buffer.slice_with_length(block.offset() as _, block_len);
+
+ let batch2 = decoder.read_record_batch(block, &data).unwrap().unwrap();
+
+ assert_eq!(batch, batch2);
+ }
+ }
+
+ #[test]
+ fn test_decimal128_alignment8_is_unaligned() {
+ const IPC_ALIGNMENT: u8 = 8;
Review Comment:
This test is equivalent to the current behavior without the changes in this PR, it's equivalent to using `pad_to_8` everywhere.
Because the Decimal128Array is insufficiently aligned, this prevents readers from doing a _true_ zero-copy read (e.g. mmapping), because they need to copy the block into a more-aligned buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546796957
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
)
.unwrap();
assert_eq!(batch, roundtrip);
}
+ #[test]
+ fn test_unaligned_throws_error_with_enforce_zero_copy() {
+ let batch = RecordBatch::try_from_iter(vec![(
+ "i32",
+ Arc::new(Int32Array::from(vec![1, 2, 3, 4])) as _,
+ )])
+ .unwrap();
+
+ let gen = IpcDataGenerator {};
+ let mut dict_tracker = DictionaryTracker::new(false);
+ let (_, encoded) = gen
+ .encoded_batch(&batch, &mut dict_tracker, &Default::default())
+ .unwrap();
+
+ let message = root_as_message(&encoded.ipc_message).unwrap();
+
+ // Construct an unaligned buffer
+ let mut buffer = MutableBuffer::with_capacity(encoded.arrow_data.len() + 1);
+ buffer.push(0_u8);
+ buffer.extend_from_slice(&encoded.arrow_data);
+ let b = Buffer::from(buffer).slice(1);
+ assert_ne!(b.as_ptr().align_offset(8), 0);
+
+ let ipc_batch = message.header_as_record_batch().unwrap();
+ let result = read_record_batch(
+ &b,
+ ipc_batch,
+ batch.schema(),
+ &Default::default(),
+ None,
+ &message.version(),
+ true,
+ );
+
+ let error = result.unwrap_err();
+ match error {
+ ArrowError::InvalidArgumentError(e) => {
+ assert!(e.contains("Misaligned"));
+ assert!(e.contains("offset from expected alignment of"));
+ }
+ _ => panic!("Expected InvalidArgumentError"),
+ }
Review Comment:
for sure - but the error message is dynamic so I figured a strict equality check was too strict - e.g. if the error message formatting or some debug impl changes, should I care about that in this test?
happy to do a strict equality check if that's generally the style in this repo!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1549541273
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
Perhaps we could do:
```rust
pub fn read_record_batch_with_options(buf: &Buffer, opts: &SomeNewOptsStruct) { ...
}
```
And then have `read_record_batch` call that new function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1550275558
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
That makes sense. I think you guys will have a vision around what goes into the struct vs what doesn't, and where throughout the codebase that struct should be passed (probably it isn't specific to read_record_batch).
To keep this PR bounded, what I'll propose for now is just make a private version, e.g. `read_record_batch2` with this extra `require_alignment: bool` arg, and keep the signature of the `pub fn read_record_batch` exactly as-is - the public one calls the private one with `false` to keep the behavior the same.
Once merged, you guys can spin up the work to figure out struct to collect the various parameters, and the `require_alignment` functionality can eventually be exposed through that options struct.
Does keeping this PR bounded in this way sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment when reading IPC, respect alignment when writing [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1551324041
##########
arrow-ipc/src/reader.rs:
##########
@@ -495,7 +524,7 @@ impl<'a> ArrayReader<'a> {
/// and copy over the data if any array data in the input `buf` is not properly aligned.
/// (Properly aligned array data will remain zero-copy.)
/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
-pub fn read_record_batch(
Review Comment:
Can we keep the doc comment on the non-private version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546793073
##########
arrow-integration-testing/src/flight_server_scenarios/integration_test.rs:
##########
@@ -308,6 +308,7 @@ async fn record_batch_from_message(
dictionaries_by_id,
None,
&message.version(),
+ false,
Review Comment:
Probably wouldn't fail - I didn't read through the full integration test but as long as it's not using Decimal128 then 8-byte alignment is sufficient, so it should succeed even with `require_alignment=true`.
Passing `false` is equivalent to the previous behavior where a copy was automatically made if the data was unaligned, so I didn't want to introduce new behavior to existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1538470815
##########
arrow-buffer/src/buffer/scalar.rs:
##########
@@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
- /// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
+ /// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
+ assert_eq!(
+ buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
+ 0,
+ "buffer is not aligned"
+ );
Review Comment:
It is checked in the from conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1550178314
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
Yeah something like that, or possibly some sort of `Decoder` struct with a member function on it...
Ideally it would be designed so that it could be shared across things like `StreamDecoder` and `FileDecoder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment when reading IPC, respect alignment when writing [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546848826
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
)
.unwrap();
assert_eq!(batch, roundtrip);
}
+ #[test]
+ fn test_unaligned_throws_error_with_enforce_zero_copy() {
+ let batch = RecordBatch::try_from_iter(vec![(
+ "i32",
+ Arc::new(Int32Array::from(vec![1, 2, 3, 4])) as _,
+ )])
+ .unwrap();
+
+ let gen = IpcDataGenerator {};
+ let mut dict_tracker = DictionaryTracker::new(false);
+ let (_, encoded) = gen
+ .encoded_batch(&batch, &mut dict_tracker, &Default::default())
+ .unwrap();
+
+ let message = root_as_message(&encoded.ipc_message).unwrap();
+
+ // Construct an unaligned buffer
+ let mut buffer = MutableBuffer::with_capacity(encoded.arrow_data.len() + 1);
+ buffer.push(0_u8);
+ buffer.extend_from_slice(&encoded.arrow_data);
+ let b = Buffer::from(buffer).slice(1);
+ assert_ne!(b.as_ptr().align_offset(8), 0);
+
+ let ipc_batch = message.header_as_record_batch().unwrap();
+ let result = read_record_batch(
+ &b,
+ ipc_batch,
+ batch.schema(),
+ &Default::default(),
+ None,
+ &message.version(),
+ true,
+ );
+
+ let error = result.unwrap_err();
+ match error {
+ ArrowError::InvalidArgumentError(e) => {
+ assert!(e.contains("Misaligned"));
+ assert!(e.contains("offset from expected alignment of"));
+ }
+ _ => panic!("Expected InvalidArgumentError"),
+ }
Review Comment:
doing strict equality checks now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546795121
##########
arrow-integration-testing/src/flight_server_scenarios/integration_test.rs:
##########
@@ -308,6 +308,7 @@ async fn record_batch_from_message(
dictionaries_by_id,
None,
&message.version(),
+ false,
Review Comment:
makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1538388341
##########
arrow-ipc/src/writer.rs:
##########
@@ -2234,4 +2270,124 @@ mod tests {
let in_batch = RecordBatch::try_new(schema, vec![values]).unwrap();
roundtrip_ensure_sliced_smaller(in_batch, 1000);
}
+
+ #[test]
+ fn test_decimal128_alignment16() {
+ const IPC_ALIGNMENT: u8 = 16;
+
+ for num_cols in 1..100 {
+ let num_rows = (num_cols * 7 + 11) % 100; // Deterministic swizzle
+
+ let mut fields = Vec::new();
+ let mut arrays = Vec::new();
+ for i in 0..num_cols {
+ let field = Field::new(&format!("col_{}", i), DataType::Decimal128(38, 10), true);
+ let array = Decimal128Array::from(vec![num_cols as i128; num_rows]);
+ fields.push(field);
+ arrays.push(Arc::new(array) as Arc<dyn Array>);
+ }
+ let schema = Schema::new(fields);
+ let batch = RecordBatch::try_new(Arc::new(schema), arrays).unwrap();
+
+ let mut writer = FileWriter::try_new_with_options(
+ Vec::new(),
+ batch.schema_ref(),
+ IpcWriteOptions::try_new(IPC_ALIGNMENT, false, MetadataVersion::V5).unwrap(),
+ )
+ .unwrap();
+ writer.write(&batch).unwrap();
+ writer.finish().unwrap();
+
+ let out: Vec<u8> = writer.into_inner().unwrap();
+
+ let buffer = Buffer::from_vec(out);
+ let trailer_start = buffer.len() - 10;
+ let footer_len =
+ read_footer_length(buffer[trailer_start..].try_into().unwrap()).unwrap();
+ let footer =
+ root_as_footer(&buffer[trailer_start - footer_len..trailer_start]).unwrap();
+
+ let schema = fb_to_schema(footer.schema().unwrap());
+ assert_eq!(&schema, batch.schema().as_ref());
+
+ let decoder =
+ FileDecoder::new(Arc::new(schema), footer.version()).with_enforce_zero_copy(true);
+
+ assert_eq!(footer.dictionaries().unwrap().len(), 0);
+
+ let batches = footer.recordBatches().unwrap();
+ assert_eq!(batches.len(), 1);
+
+ let block = batches.get(0);
+ let block_len = block.bodyLength() as usize + block.metaDataLength() as usize;
+ let data = buffer.slice_with_length(block.offset() as _, block_len);
+
+ let batch2 = decoder.read_record_batch(block, &data).unwrap().unwrap();
+
+ assert_eq!(batch, batch2);
+ }
+ }
+
+ #[test]
+ fn test_decimal128_alignment8_is_unaligned() {
+ const IPC_ALIGNMENT: u8 = 8;
Review Comment:
This test is equivalent to the current behavior without the changes in this PR, it's equivalent to using `pad_to_8` everywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1538393062
##########
arrow-buffer/src/buffer/scalar.rs:
##########
@@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
- /// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
+ /// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
+ assert_eq!(
+ buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
+ 0,
+ "buffer is not aligned"
+ );
Review Comment:
This was not previously checked even though the docs said it was checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1540430485
##########
arrow-buffer/src/buffer/scalar.rs:
##########
@@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
- /// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
+ /// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
+ assert_eq!(
+ buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
+ 0,
+ "buffer is not aligned"
+ );
Review Comment:
ah ok, great catch! will remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment when reading IPC, respect alignment when writing [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1551324041
##########
arrow-ipc/src/reader.rs:
##########
@@ -495,7 +524,7 @@ impl<'a> ArrayReader<'a> {
/// and copy over the data if any array data in the input `buf` is not properly aligned.
/// (Properly aligned array data will remain zero-copy.)
/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
-pub fn read_record_batch(
Review Comment:
Can we keep the doc comment on the non-private version?
Edit: I took the liberty of doing this to get this in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546793887
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
Review Comment:
yes! the test below is literally this exact with true instead of false, and that test checks that it throws
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1550489729
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
okay pushed the above - lmk if you would prefer a different approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#issuecomment-2019308748
I'm travelling for the next few days, but will try to get to this on my return if nobody else gets there first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optionally require alignment requirements when reading IPC, respect alignment when writing [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1550275558
##########
arrow-ipc/src/reader.rs:
##########
@@ -428,14 +485,24 @@ impl<'a> ArrayReader<'a> {
}
}
-/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`
+/// Creates a record batch from binary data using the `crate::RecordBatch` indexes and the `Schema`.
+///
+/// If `require_alignment` is true, this function will return an error if any array data in the
+/// input `buf` is not properly aligned.
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build`] to construct [`arrow_data::ArrayData`].
+///
+/// If `require_alignment` is false, this function will automatically allocate a new aligned buffer
+/// and copy over the data if any array data in the input `buf` is not properly aligned.
+/// (Properly aligned array data will remain zero-copy.)
+/// Under the hood it will use [`arrow_data::ArrayDataBuilder::build_aligned`] to construct [`arrow_data::ArrayData`].
pub fn read_record_batch(
buf: &Buffer,
batch: crate::RecordBatch,
schema: SchemaRef,
dictionaries_by_id: &HashMap<i64, ArrayRef>,
projection: Option<&[usize]>,
metadata: &MetadataVersion,
+ require_alignment: bool,
Review Comment:
That makes sense. I think you guys will have a vision around what goes into the struct vs what doesn't, and where throughout the codebase that struct should be passed (probably it isn't specific to read_record_batch).
To keep this PR bounded, what I'll propose for now is just make a private version, e.g. `read_record_batch2` with this extra `require_alignment: bool` arg, and keep the signature of the `pub fn read_record_batch` exactly as-is - the public one calls the private one with `false` to keep the behavior the same.
Once merged, you guys can spin up the work to figure out the struct to collect the various parameters, where that struct should be used, etc. and the `require_alignment` functionality can eventually be exposed through that struct.
Does keeping this PR bounded in this way sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "hzuo (via GitHub)" <gi...@apache.org>.
hzuo commented on code in PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#discussion_r1546793887
##########
arrow-ipc/src/reader.rs:
##########
@@ -1801,11 +1878,56 @@ mod tests {
&Default::default(),
None,
&message.version(),
+ false,
Review Comment:
yes! the test below is literally this exact test, but with true instead of false, and that test checks that it throws
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Respect alignment requirements when writing IPC data [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5554:
URL: https://github.com/apache/arrow-rs/pull/5554#issuecomment-2033100560
so close!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org