You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/11 01:32:17 UTC

[GitHub] [arrow-rs] viirya opened a new pull request, #2040: Truncate IPC record batch

viirya opened a new pull request, #2040:
URL: https://github.com/apache/arrow-rs/pull/2040

   # Which issue does this PR close?
   
   <!---
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #1528.
   
   # Rationale for this change
    
    <!---
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   # What changes are included in this PR?
   
   <!---
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are there any user-facing changes?
   
   
   <!---
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184906589

   I don't find C++ implementation does truncation for ListArray, StructArray, but maybe I miss it when reading the code. I will re-check this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184904344

   I think there is more to be done in this vein, e.g. ListArray, StructArray, etc... but this is a good step forward. Thank you 🙂


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919402702


##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];
+            for (idx, x) in offsets.iter().enumerate() {
+                new_offsets[idx] = x - start_offset;
+            }
+
+            Buffer::from_slice_ref(&new_offsets)

Review Comment:
   Doesn't `BufferBuilder.append_slice` also copy slice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921477720


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);
+
+            RecordBatch::try_new(Arc::new(schema), vec![Arc::new(a), Arc::new(b)])
+                .unwrap()
+        }
+
+        let big_record_batch = create_batch(65536);
+
+        let length = 5;
+        let small_record_batch = create_batch(length);
+
+        let offset = 2;
+        let record_batch_slice = big_record_batch.slice(offset, length);
+        assert!(
+            serialize_with_truncate(&big_record_batch).len()
+                > serialize_with_truncate(&small_record_batch).len()
+        );
+        assert_eq!(
+            serialize_with_truncate(&small_record_batch).len(),
+            serialize_with_truncate(&record_batch_slice).len()
+        );
+
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            small_record_batch
+        );
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            record_batch_slice
         );
+
+        // Disable truncation
+        assert!(
+            serialize_without_truncate(&record_batch_slice).len()
+                > serialize_without_truncate(&small_record_batch).len()
+        );
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch_with_nulls() {
+        fn create_batch() -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, true),
+                Field::new("b", DataType::Utf8, true),
+            ]);
+
+            let a = Int32Array::from(vec![Some(1), None, Some(1), None, Some(1)]);
+            let b = StringArray::from(vec![None, Some("a"), Some("a"), None, Some("a")]);

Review Comment:
   This PR fails with the above change, which would suggest something isn't working quite right



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r917600451


##########
arrow/src/ipc/writer.rs:
##########
@@ -1507,4 +1702,60 @@ mod tests {
             IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
         );
     }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        pub fn serialize(record: &RecordBatch) -> Vec<u8> {

Review Comment:
   No need to be. I copied from the test provided in the issue. Let me remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921504560


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);

Review Comment:
   Oh, originally all elements are same in two arrays(int, string). So `small_record_batch` is equal to deserialized `record_batch_slice`. The change makes array elements different, so `small_record_batch` is different to deserialized `record_batch_slice` (e.g. [2, 3, 4, 5, 6] to [0, 1, 2, 3, 4]).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] JasonLi-cn commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
JasonLi-cn commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r920765737


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1004,88 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        let sliced_null_buffer = if write_options.enable_truncate_array {
+            null_buffer.bit_slice(array_data.offset(), array_data.len())
+        } else {
+            null_buffer
+        };
+        offset = write_buffer(sliced_null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   I think there should be processed like this: 
   ```rust
       if has_validity_bitmap(array_data.data_type(), write_options) {
           // write null buffer if exists
           let null_buffer = match array_data.null_buffer() {
               None => {
                   // create a buffer and fill it with valid bits
                   let num_bytes = bit_util::ceil(num_rows, 8);
                   let buffer = MutableBuffer::new(num_bytes);
                   let buffer = buffer.with_bitset(num_bytes, true);
                   buffer.into()
               }
               Some(buffer) => {
                   if write_options.enable_truncate_array {
                       buffer.bit_slice(array_data.offset(), array_data.len())
                   } else {
                       buffer.clone()
                   }
               }
           };
   
           offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
       }
   ```
   Slice of RecordBatch which has no null_buffer may case Panic, for example: record_batch.num_rows() = 100,  let rb_slice = record_batch.slice(64, 36), process rb_slice will case Panic("the offset of the new Buffer cannot exceed the existing length") when call null_buffer.bit_slice(array_data.offset(), array_data.len()). 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184793174

   I was looking for how the reader knows how to ignore it. But I didn't find it. Then I realized that previously the writer wrote the entire buffers. For the reader, it sets offsets from zero and reads the buffers from incorrect positions.
   
   So I think it is a bug that sliced record batch cannot be read correctly after IPC. We can reproduce it:
   
   ```rust
   fn create_batch() -> RecordBatch {
     let schema = Schema::new(vec![
       Field::new("a", DataType::Int32, true),
       Field::new("b", DataType::Utf8, true),
     ]);
   
     let a = Int32Array::from(vec![1, 2, 3, 4, 5]);
     let b = StringArray::from(vec!["a", "b", "c", "d", "e"]);
   
     RecordBatch::try_new(Arc::new(schema), vec![Arc::new(a), Arc::new(b)])
       .unwrap()
   }
   
   let big_record_batch = create_batch();
   
   let offset = 2;
   let record_batch_slice = big_record_batch.slice(offset, 3);
   ```
   
   Original record batches:
   ```
   RecordBatch { schema: Schema { fields: [Field { name: "a", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "b", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: fal
   se, metadata: None }], metadata: {} }, columns: [PrimitiveArray<Int32>
   [                                                                                                                  
     1,
     2,                                                                                                               
     3,
     4,                                                                                                               
     5,
   ], StringArray 
   [
     "a",                                                                                                             
     "b",
     "c",                                                                                                             
     "d",
     "e",         
   ]], row_count: 5 }
   RecordBatch { schema: Schema { fields: [Field { name: "a", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "b", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: fal
   se, metadata: None }], metadata: {} }, columns: [PrimitiveArray<Int32>
   [                                                                                                                  
     3,
     4,
     5,
   ], StringArray
   [
     "c",
     "d",
     "e",
   ]], row_count: 3 }
   ```
   
   
   Record batches after IPC serialization/deserialzation:
   
   ```
   RecordBatch { schema: Schema { fields: [Field { name: "a", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "b", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: fal
   se, metadata: None }], metadata: {} }, columns: [PrimitiveArray<Int32>                       
   [
     1,                                                                                                               
     2,
     3,           
     4,
     5,                                                                                                               
   ], StringArray
   [                                                                                                                                                                                                                                      
     "a",
     "b",         
     "c",
     "d",                                                                                                             
     "e",
   ]], row_count: 5 }                                                                                                                                                                                                                     
   RecordBatch { schema: Schema { fields: [Field { name: "a", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "b", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: fal
   se, metadata: None }], metadata: {} }, columns: [PrimitiveArray<Int32>
   [
     null,                                                                                                            
     null,
     5,                                                                                                               
   ], StringArray
   [              
     "a",
     "b",                                                                                                             
     "c",
   ]], row_count: 3 } 
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921477386


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);
+
+            RecordBatch::try_new(Arc::new(schema), vec![Arc::new(a), Arc::new(b)])
+                .unwrap()
+        }
+
+        let big_record_batch = create_batch(65536);
+
+        let length = 5;
+        let small_record_batch = create_batch(length);
+
+        let offset = 2;
+        let record_batch_slice = big_record_batch.slice(offset, length);
+        assert!(
+            serialize_with_truncate(&big_record_batch).len()
+                > serialize_with_truncate(&small_record_batch).len()
+        );
+        assert_eq!(
+            serialize_with_truncate(&small_record_batch).len(),
+            serialize_with_truncate(&record_batch_slice).len()
+        );
+
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            small_record_batch
+        );
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            record_batch_slice
         );
+

Review Comment:
   ```suggestion
           assert_eq!(
               deserialize(serialize_without_truncate(&record_batch_slice)),
               record_batch_slice
           );
   ```
   Or something, basically I think the non-sliced logic was and still is wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold merged pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] codecov-commenter commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1179873238

   # [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#2040](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (71d1dd6) into [master](https://codecov.io/gh/apache/arrow-rs/commit/330505c98c9db4d103b4965e8bd55065961dada7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (330505c) will **decrease** coverage by `0.01%`.
   > The diff coverage is `88.67%`.
   
   > :exclamation: Current head 71d1dd6 differs from pull request most recent head bd63562. Consider uploading reports for the commit bd63562 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #2040      +/-   ##
   ==========================================
   - Coverage   83.55%   83.53%   -0.02%     
   ==========================================
     Files         222      222              
     Lines       58230    58317      +87     
   ==========================================
   + Hits        48656    48718      +62     
   - Misses       9574     9599      +25     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [arrow/src/array/mod.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L21vZC5ycw==) | `100.00% <ø> (ø)` | |
   | [arrow/src/datatypes/datatype.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9kYXRhdHlwZS5ycw==) | `64.05% <30.00%> (-1.63%)` | :arrow_down: |
   | [arrow/src/ipc/writer.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2lwYy93cml0ZXIucnM=) | `82.05% <82.40%> (+0.26%)` | :arrow_up: |
   | [arrow/src/array/builder/generic\_binary\_builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIvZ2VuZXJpY19iaW5hcnlfYnVpbGRlci5ycw==) | `83.78% <100.00%> (+26.64%)` | :arrow_up: |
   | [arrow/src/array/builder/generic\_list\_builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIvZ2VuZXJpY19saXN0X2J1aWxkZXIucnM=) | `95.09% <100.00%> (-1.61%)` | :arrow_down: |
   | [arrow/src/array/builder/generic\_string\_builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIvZ2VuZXJpY19zdHJpbmdfYnVpbGRlci5ycw==) | `92.13% <100.00%> (+10.08%)` | :arrow_up: |
   | [...row/src/array/builder/string\_dictionary\_builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIvc3RyaW5nX2RpY3Rpb25hcnlfYnVpbGRlci5ycw==) | `90.64% <0.00%> (-0.72%)` | :arrow_down: |
   | [parquet/src/encodings/encoding.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvZW5jb2RpbmdzL2VuY29kaW5nLnJz) | `93.43% <0.00%> (-0.20%)` | :arrow_down: |
   | [arrow/src/array/data.rs](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2RhdGEucnM=) | `84.78% <0.00%> (+0.24%)` | :arrow_up: |
   | ... and [2 more](https://codecov.io/gh/apache/arrow-rs/pull/2040/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [330505c...bd63562](https://codecov.io/gh/apache/arrow-rs/pull/2040?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] HaoYang670 commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
HaoYang670 commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r917592629


##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,145 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    if spec == &BufferSpec::AlwaysNull {
+        false
+    } else {
+        array_offset != 0 || min_length < buffer.len()
+    }
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize

Review Comment:
   Is this corrent when `array_data.offset() > 0`?



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,145 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    if spec == &BufferSpec::AlwaysNull {
+        false
+    } else {
+        array_offset != 0 || min_length < buffer.len()
+    }
+}

Review Comment:
   nit: (not tested)
   ```rust
   spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
   ```



##########
arrow/src/ipc/writer.rs:
##########
@@ -1507,4 +1702,60 @@ mod tests {
             IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
         );
     }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        pub fn serialize(record: &RecordBatch) -> Vec<u8> {

Review Comment:
   Why do we need this function to be public?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919245498


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   But doesn't the truncation remove the offset from the array data, and instead materialize it into the underlying buffers. As a result I think this will result in incorrect behaviour as whilst we do materialize the offset into the values buffers, we don't do the same for the null buffer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919245498


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   But doesn't the truncation remove the offset from the array data, and materialize it into the underlying buffers. As a result I think this will result in incorrect behaviour as whilst we do materialize the offset into the values buffers, we don't do the same for the null buffer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919221579


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   Yea, although it is truncated at the offset, but the length is not considered. As null buffer is bit encoded, and different to other buffers here, it cannot be simply truncated by the length. Not aware of there is `bit_slice` that can be used.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r917602896


##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,145 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    if spec == &BufferSpec::AlwaysNull {
+        false
+    } else {
+        array_offset != 0 || min_length < buffer.len()
+    }
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize

Review Comment:
   Yes. `value_offsets` has considered `array_data`'s `offset`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919419568


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();

Review Comment:
   It sounds reasonable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919257679


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   `null_buffer.as_slice()` already takes the slice from the offset from null buffer. I think it is you meant materializing the buffers (null buffer and value buffers)? The difference now is this also truncate the length from value buffers but not for null buffer (we can do it by using `bit_slice` then).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] alamb commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1182365207

   > Yes, flight data also uses IPC to encode record batch. This also closes https://github.com/apache/arrow-rs/issues/208 too.
   
   Updated the PR description to match


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921505696


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);

Review Comment:
   Actually the check between `small_record_batch` and deserialized `record_batch_slice` is not necessary. Let me revise the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919235111


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();
+    if matches!(
+        data_type,
+        DataType::Binary | DataType::LargeBinary | DataType::Utf8 | DataType::LargeUtf8
+    ) {
+        let total_bytes = get_total_bytes(array_data);
+        let value_buffer = &array_data.buffers()[1];
+        if buffer_need_truncate(
+            array_data.offset(),
+            value_buffer,
+            &BufferSpec::VariableWidth,
+            total_bytes,
+        ) {
+            // Rebase offsets and truncate values
+            let new_offsets = get_zero_based_value_offsets(array_data);
+            offset = write_buffer(new_offsets.as_slice(), buffers, arrow_data, offset);
+
+            let byte_offset = get_buffer_offset(array_data);
+            let buffer_length = min(total_bytes, value_buffer.len() - byte_offset);
+            let buffer_slice =
+                &value_buffer.as_slice()[byte_offset..(byte_offset + buffer_length)];
+            offset = write_buffer(buffer_slice, buffers, arrow_data, offset);
+        } else {
+            array_data.buffers().iter().for_each(|buffer| {
+                offset = write_buffer(buffer.as_slice(), buffers, arrow_data, offset);
+            });
+        }
+    } else if DataType::is_numeric(data_type)
+        || DataType::is_temporal(data_type)
+        || matches!(array_data.data_type(), DataType::FixedSizeBinary(_))

Review Comment:
   Hmm, yea, seems so. Rechecked C++ implementation, it truncates DictionaryArray keys: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L524-L527



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1182016689

   Yes, flight data also uses IPC to encode record batch. This also closes #208 too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184905834

   Thanks @alamb @tustvold @JasonLi-cn @HaoYang670 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] alamb commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1185560056

   > I think there is more to be done in this vein, e.g. ListArray, StructArray, etc... but this is a good step forward. Thank you 🙂
   
   
   @tustvold  is the remaining work recorded anywhere? I am happy to write up a ticket but I don't understand the issue well enough to do so at this moment
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r920772923


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1004,88 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        let sliced_null_buffer = if write_options.enable_truncate_array {
+            null_buffer.bit_slice(array_data.offset(), array_data.len())
+        } else {
+            null_buffer
+        };
+        offset = write_buffer(sliced_null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   Good catch. The null buffer created is for `num_rows` so it is already truncated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921478342


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);

Review Comment:
   ```suggestion
               let a = Int32Array::from_iter_values(0..rows as i32);
               let b = StringArray::from_iter_values((0..rows).map(|i| i.to_string()));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184798602

   Perhaps we should make it non-optional then, I had thought the offset was separately preserved, given it isn't, there doesn't seem to be a reason to have the option


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] alamb commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1182008784

   ❤️  -- thank you @viirya 
   
   Note we have some code in IOx to workaround the lack of this feature (described in https://github.com/influxdata/influxdb_iox/issues/1133): 
    https://github.com/influxdata/influxdb_iox/blob/3592aa52d815e861ac14031186ba557c9290dfaf/arrow_util/src/optimize.rs#L97-L141
   
   I wonder if this PR also closes https://github.com/apache/arrow-rs/issues/208?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919245498


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   But doesn't the truncation remove the offset from the array data, and instead materialize it into the underlying buffers. As a result I think this will result in incorrect behaviour as whilst we do materialize the offset into the values buffers, we don't do the same for the null buffer?
   
   > Not aware of there is bit_slice that can be used.
   
   https://docs.rs/arrow/latest/arrow/buffer/struct.Buffer.html#method.bit_slice



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919404802


##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];
+            for (idx, x) in offsets.iter().enumerate() {
+                new_offsets[idx] = x - start_offset;
+            }
+
+            Buffer::from_slice_ref(&new_offsets)

Review Comment:
   Yes, but this is copying twice - once into the Vec and once here. If you used BufferBuilder instead of Vec you would eliminate one of these copies



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921477386


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);
+
+            RecordBatch::try_new(Arc::new(schema), vec![Arc::new(a), Arc::new(b)])
+                .unwrap()
+        }
+
+        let big_record_batch = create_batch(65536);
+
+        let length = 5;
+        let small_record_batch = create_batch(length);
+
+        let offset = 2;
+        let record_batch_slice = big_record_batch.slice(offset, length);
+        assert!(
+            serialize_with_truncate(&big_record_batch).len()
+                > serialize_with_truncate(&small_record_batch).len()
+        );
+        assert_eq!(
+            serialize_with_truncate(&small_record_batch).len(),
+            serialize_with_truncate(&record_batch_slice).len()
+        );
+
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            small_record_batch
+        );
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            record_batch_slice
         );
+

Review Comment:
   ```suggestion
           assert_eq!(
               deserialize(serialize_without_truncate(&record_batch_slice)),
               record_batch_slice
           );
   ```
   Or something, basically I think the previous logic was wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r918930065


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();

Review Comment:
   I wonder if we should add an option to IpcWriteOptions to enable/disable this behaviour. I could see in some cases the rewriting not being worthwhile



##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   This seems suspect to me, we are truncating the values, but don't appear to be doing the same for the null mask?



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];
+            for (idx, x) in offsets.iter().enumerate() {
+                new_offsets[idx] = x - start_offset;
+            }
+
+            Buffer::from_slice_ref(&new_offsets)
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+
+            let start_offset = offsets[0];

Review Comment:
   It might be nice to extract the copy-pasted logic here into a generic function on `O: OffsetSize`



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];
+            for (idx, x) in offsets.iter().enumerate() {
+                new_offsets[idx] = x - start_offset;
+            }
+
+            Buffer::from_slice_ref(&new_offsets)

Review Comment:
   Using BufferBuilder will be faster as it will avoid a copy



##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();
+    if matches!(
+        data_type,
+        DataType::Binary | DataType::LargeBinary | DataType::Utf8 | DataType::LargeUtf8
+    ) {
+        let total_bytes = get_total_bytes(array_data);
+        let value_buffer = &array_data.buffers()[1];
+        if buffer_need_truncate(
+            array_data.offset(),
+            value_buffer,
+            &BufferSpec::VariableWidth,
+            total_bytes,
+        ) {
+            // Rebase offsets and truncate values
+            let new_offsets = get_zero_based_value_offsets(array_data);
+            offset = write_buffer(new_offsets.as_slice(), buffers, arrow_data, offset);
+
+            let byte_offset = get_buffer_offset(array_data);
+            let buffer_length = min(total_bytes, value_buffer.len() - byte_offset);
+            let buffer_slice =
+                &value_buffer.as_slice()[byte_offset..(byte_offset + buffer_length)];
+            offset = write_buffer(buffer_slice, buffers, arrow_data, offset);
+        } else {
+            array_data.buffers().iter().for_each(|buffer| {
+                offset = write_buffer(buffer.as_slice(), buffers, arrow_data, offset);
+            });
+        }
+    } else if DataType::is_numeric(data_type)
+        || DataType::is_temporal(data_type)
+        || matches!(array_data.data_type(), DataType::FixedSizeBinary(_))

Review Comment:
   I think this could also support DictionaryArray without modification?



##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();
+    if matches!(
+        data_type,
+        DataType::Binary | DataType::LargeBinary | DataType::Utf8 | DataType::LargeUtf8
+    ) {
+        let total_bytes = get_total_bytes(array_data);
+        let value_buffer = &array_data.buffers()[1];
+        if buffer_need_truncate(
+            array_data.offset(),
+            value_buffer,
+            &BufferSpec::VariableWidth,
+            total_bytes,
+        ) {
+            // Rebase offsets and truncate values
+            let new_offsets = get_zero_based_value_offsets(array_data);
+            offset = write_buffer(new_offsets.as_slice(), buffers, arrow_data, offset);

Review Comment:
   Definitely something that could be left to another PR, but I do wonder if there might be some way to write the new offsets directly without buffering them first. I'm not very familiar with the IPC code so not sure how feasible this may be



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {

Review Comment:
   ```suggestion
   fn get_binary_buffer_len(array_data: &ArrayData) -> usize {
   ```
   Or something to make a bit clear what it is the total of?



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];

Review Comment:
   I don't think we need to zero-initialize here



##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {

Review Comment:
   ```suggestion
   fn get_buffer_element_width(spec: &BufferSpec) -> usize {
   ```



##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();
+    if matches!(
+        data_type,
+        DataType::Binary | DataType::LargeBinary | DataType::Utf8 | DataType::LargeUtf8
+    ) {
+        let total_bytes = get_total_bytes(array_data);
+        let value_buffer = &array_data.buffers()[1];
+        if buffer_need_truncate(
+            array_data.offset(),
+            value_buffer,
+            &BufferSpec::VariableWidth,
+            total_bytes,
+        ) {
+            // Rebase offsets and truncate values
+            let new_offsets = get_zero_based_value_offsets(array_data);
+            offset = write_buffer(new_offsets.as_slice(), buffers, arrow_data, offset);
+
+            let byte_offset = get_buffer_offset(array_data);
+            let buffer_length = min(total_bytes, value_buffer.len() - byte_offset);
+            let buffer_slice =
+                &value_buffer.as_slice()[byte_offset..(byte_offset + buffer_length)];
+            offset = write_buffer(buffer_slice, buffers, arrow_data, offset);
+        } else {
+            array_data.buffers().iter().for_each(|buffer| {
+                offset = write_buffer(buffer.as_slice(), buffers, arrow_data, offset);
+            });
+        }
+    } else if DataType::is_numeric(data_type)
+        || DataType::is_temporal(data_type)
+        || matches!(array_data.data_type(), DataType::FixedSizeBinary(_))
+    {
+        // Truncate values
+        let layout = layout(data_type);
+
+        array_data

Review Comment:
   This code could theoretically be simpler if it made the valid assertion that all these arrays only have a single buffer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919426839


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);
     }
 
-    array_data.buffers().iter().for_each(|buffer| {
-        offset = write_buffer(buffer, buffers, arrow_data, offset);
-    });
+    let data_type = array_data.data_type();

Review Comment:
   Added an option.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919340795


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   There are two offsets, the offset within the null buffer which `as_slice` applies, and the offset from the ArrayData which `as_slice` cannot possibly apply as it has no idea what it is (and it is an offset in bits and not bytes).
   
   I would like to see a test of a nullable sliced array, it is possible I'm mistaken, but I am pretty certain this will break



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919223229


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   This seems not to be an (serious) issue? Null buffer is relatively small. One issue is the length of null buffer after IPC is not matched. But I don't see that we really care about it in current code?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919405642


##########
arrow/src/ipc/writer.rs:
##########
@@ -861,6 +863,141 @@ fn has_validity_bitmap(data_type: &DataType, write_options: &IpcWriteOptions) ->
     }
 }
 
+/// Whether to truncate the buffer
+#[inline]
+fn buffer_need_truncate(
+    array_offset: usize,
+    buffer: &Buffer,
+    spec: &BufferSpec,
+    min_length: usize,
+) -> bool {
+    spec != &BufferSpec::AlwaysNull && (array_offset != 0 || min_length < buffer.len())
+}
+
+/// Returns byte width for a buffer spec. Only for `BufferSpec::FixedWidth`.
+#[inline]
+fn get_buffer_byte_width(spec: &BufferSpec) -> usize {
+    match spec {
+        BufferSpec::FixedWidth { byte_width } => *byte_width,
+        _ => 0,
+    }
+}
+
+/// Returns the number of total bytes in base binary arrays.
+fn get_total_bytes(array_data: &ArrayData) -> usize {
+    if array_data.is_empty() {
+        return 0;
+    }
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeBinary => {
+            let array: LargeBinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::Utf8 => {
+            let array: StringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        DataType::LargeUtf8 => {
+            let array: LargeStringArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            (offsets[array_data.len()] - offsets[0]) as usize
+        }
+        _ => unreachable!(),
+    }
+}
+
+/// Rebase value offsets for given ArrayData to zero-based.
+fn get_zero_based_value_offsets(array_data: &ArrayData) -> Buffer {
+    match array_data.data_type() {
+        DataType::Binary => {
+            let array: BinaryArray = array_data.clone().into();
+            let offsets = array.value_offsets();
+            let start_offset = offsets[0];
+
+            let mut new_offsets = vec![0_i32; array_data.len() + 1];
+            for (idx, x) in offsets.iter().enumerate() {
+                new_offsets[idx] = x - start_offset;
+            }
+
+            Buffer::from_slice_ref(&new_offsets)

Review Comment:
   Ah, I see.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921475540


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);
+
+            RecordBatch::try_new(Arc::new(schema), vec![Arc::new(a), Arc::new(b)])
+                .unwrap()
+        }
+
+        let big_record_batch = create_batch(65536);
+
+        let length = 5;
+        let small_record_batch = create_batch(length);
+
+        let offset = 2;
+        let record_batch_slice = big_record_batch.slice(offset, length);
+        assert!(
+            serialize_with_truncate(&big_record_batch).len()
+                > serialize_with_truncate(&small_record_batch).len()
+        );
+        assert_eq!(
+            serialize_with_truncate(&small_record_batch).len(),
+            serialize_with_truncate(&record_batch_slice).len()
+        );
+
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            small_record_batch
+        );
+        assert_eq!(
+            deserialize(serialize_with_truncate(&record_batch_slice)),
+            record_batch_slice
         );
+
+        // Disable truncation
+        assert!(
+            serialize_without_truncate(&record_batch_slice).len()
+                > serialize_without_truncate(&small_record_batch).len()
+        );
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch_with_nulls() {
+        fn create_batch() -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, true),
+                Field::new("b", DataType::Utf8, true),
+            ]);
+
+            let a = Int32Array::from(vec![Some(1), None, Some(1), None, Some(1)]);
+            let b = StringArray::from(vec![None, Some("a"), Some("a"), None, Some("a")]);

Review Comment:
   ```suggestion
               let a = Int32Array::from_iter_values(0..rows as i32);
               let b = StringArray::from_iter_values((0..rows).map(|i| i.to_string()));
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r921478821


##########
arrow/src/ipc/writer.rs:
##########
@@ -1501,10 +1690,151 @@ mod tests {
     #[test]
     fn test_write_union_file_v4_v5() {
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V4).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V4, false).unwrap(),
         );
         write_union_file(
-            IpcWriteOptions::try_new(8, false, MetadataVersion::V5).unwrap(),
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap(),
+        );
+    }
+
+    fn serialize_with_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options = IpcWriteOptions::default();
+        serialize(record, write_options)
+    }
+
+    fn serialize_without_truncate(record: &RecordBatch) -> Vec<u8> {
+        let write_options =
+            IpcWriteOptions::try_new(8, false, MetadataVersion::V5, false).unwrap();
+        serialize(record, write_options)
+    }
+
+    fn serialize(record: &RecordBatch, write_options: IpcWriteOptions) -> Vec<u8> {
+        let buffer: Vec<u8> = Vec::new();
+        let mut stream_writer =
+            StreamWriter::try_new_with_options(buffer, &record.schema(), write_options)
+                .unwrap();
+        stream_writer.write(record).unwrap();
+        stream_writer.finish().unwrap();
+        stream_writer.into_inner().unwrap()
+    }
+
+    fn deserialize(bytes: Vec<u8>) -> RecordBatch {
+        let mut stream_reader =
+            ipc::reader::StreamReader::try_new(std::io::Cursor::new(bytes), None)
+                .unwrap();
+        stream_reader.next().unwrap().unwrap()
+    }
+
+    #[test]
+    fn truncate_ipc_record_batch() {
+        fn create_batch(rows: usize) -> RecordBatch {
+            let schema = Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("b", DataType::Utf8, false),
+            ]);
+
+            let a = Int32Array::from(vec![1; rows]);
+            let b = StringArray::from(vec!["a"; rows]);

Review Comment:
   This PR fails with the above change, which would suggest something isn't quite right



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184794445

   That's said that we need to truncate it without the option. I don't find C++ implementation has an option for that actually.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184889814

   Unrelated error on Windows CI:
   
   ```
   error: error: Invalid value for '<+toolchain>': Toolchain overrides must begin with '+'
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] viirya commented on a diff in pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#discussion_r919372229


##########
arrow/src/ipc/writer.rs:
##########
@@ -894,12 +1031,66 @@ fn write_array_data(
             Some(buffer) => buffer.clone(),
         };
 
-        offset = write_buffer(&null_buffer, buffers, arrow_data, offset);
+        offset = write_buffer(null_buffer.as_slice(), buffers, arrow_data, offset);

Review Comment:
   Yea, figured it out now. You're right. Array data's offset is not considered for null buffer yet. I'm fixing it and adding related test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1185616150

   Filed #2080 to track follow up work, I personally think we should accelerate efforts to remove offset from ArrayData, as opposed to continuing to work around it in various places


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on pull request #2040: Truncate IPC record batch

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2040:
URL: https://github.com/apache/arrow-rs/pull/2040#issuecomment-1184778315

   Apologies I'm trying to review this, but I'm extremely confused as to how this has been and is working, I can't find where the IPC writer makes use of the ArrayData offset, and am therefore at a loss as to how slicing has historically been being preserved. Could you possibly point me at the relevant code, as far as I can tell it previously wrote the entirety of the buffers, but I can't work out how the reader then knows to ignore most of this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org