You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "fabianmurariu (via GitHub)" <gi...@apache.org> on 2025/12/10 12:42:05 UTC
[I] Wrong data in IPC format for Non zero offset start for LargeListArray [arrow]
fabianmurariu opened a new issue, #48428:
URL: https://github.com/apache/arrow/issues/48428
### Describe the bug, including details regarding any error messages, version, and platform.
When writing to Arrow IPC a List that has an offset not starting at 0, upon reading back the data the offset is not preserved so the List contains wrong data.
Replicated [here](https://github.com/fabianmurariu/arrow_issues) with simple test, working for offset 0..2 failing for offset 2..4
### Component(s)
Format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong data in IPC format for Non zero offset start for LargeListArray with Utf8View values [arrow]
Posted by "fabianmurariu (via GitHub)" <gi...@apache.org>.
fabianmurariu commented on issue #48428:
URL: https://github.com/apache/arrow/issues/48428#issuecomment-3636973457
It looks like this only fails for StringView
```rust
#[test]
fn encode_large_lists_non_zero_offset() { //PASS
let val_inner = Field::new_list_field(DataType::UInt32, true);
let val_list_field = Field::new("val", DataType::LargeList(Arc::new(val_inner)), false);
let schema = Arc::new(Schema::new(vec![val_list_field]));
let values = Arc::new(generate_list_data::<i64>());
let in_batch = RecordBatch::try_new(schema, vec![values])
.unwrap()
.slice(999, 1);
let out_batch = deserialize_file(serialize_file(&in_batch));
assert_eq!(in_batch, out_batch);
}
#[test]
fn encode_large_lists_string_non_zero_offset() { //PASS
let val_inner = Field::new_list_field(DataType::Utf8, true);
let val_list_field = Field::new("val", DataType::LargeList(Arc::new(val_inner)), false);
let schema = Arc::new(Schema::new(vec![val_list_field]));
let values = Arc::new(generate_string_list_data::<i64>());
let in_batch = RecordBatch::try_new(schema, vec![values])
.unwrap()
.slice(999, 1);
let out_batch = deserialize_file(serialize_file(&in_batch));
assert_eq!(in_batch, out_batch);
}
#[test]
fn encode_large_list_string_view_non_zero_offset() { // FAIL
let val_inner = Field::new_list_field(DataType::Utf8View, true);
let val_list_field = Field::new("val", DataType::LargeList(Arc::new(val_inner)), false);
let schema = Arc::new(Schema::new(vec![val_list_field]));
let values = Arc::new(generate_utf8view_list_data::<i64>());
let in_batch = RecordBatch::try_new(schema, vec![values])
.unwrap()
.slice(999, 1);
dbg!(&in_batch);
let out_batch = deserialize_file(serialize_file(&in_batch));
assert_eq!(in_batch, out_batch);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong data in IPC format for Non zero offset start for LargeListArray with StringView values [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #48428: Wrong data in IPC format for Non zero offset start for LargeListArray with StringView values
URL: https://github.com/apache/arrow/issues/48428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org