You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/06 17:47:38 UTC
[GitHub] [arrow-rs] helgikrs opened a new pull request #1140: feat(ipc): support for reading union arrays through IPC
helgikrs opened a new pull request #1140:
URL: https://github.com/apache/arrow-rs/pull/1140
# Which issue does this PR close?
Together with #1135, closes #654.
# What changes are included in this PR?
With this change, IPC serialized arrow data containing unions can be deserialized successfully. This PR includes test cases round-tripping unions through the IPC writer and reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #1140: feat(ipc): support for reading union arrays through IPC
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1140:
URL: https://github.com/apache/arrow-rs/pull/1140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] paddyhoran commented on pull request #1140: feat(ipc): support for reading union arrays through IPC
Posted by GitBox <gi...@apache.org>.
paddyhoran commented on pull request #1140:
URL: https://github.com/apache/arrow-rs/pull/1140#issuecomment-1007431548
Thank you very much for this @helgikrs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #1140: feat(ipc): support for reading union arrays through IPC
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1140:
URL: https://github.com/apache/arrow-rs/pull/1140#discussion_r779901872
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -1183,18 +1245,48 @@ mod tests {
false,
)]));
- let batch = RecordBatch::try_new(schema.clone(), vec![struct_array]).unwrap();
+ let batch = RecordBatch::try_new(schema, vec![struct_array]).unwrap();
- let mut buf = Vec::new();
- let mut writer = ipc::writer::FileWriter::try_new(&mut buf, &schema).unwrap();
- writer.write(&batch).unwrap();
- writer.finish().unwrap();
- drop(writer);
+ assert_eq!(batch, roundtrip_ipc(&batch));
+ }
+
+ fn check_union_with_builder(mut builder: UnionBuilder) {
+ builder.append::<datatypes::Int32Type>("a", 1).unwrap();
+ builder.append_null().unwrap();
+ builder.append::<datatypes::Float64Type>("c", 3.0).unwrap();
+ builder.append::<datatypes::Int32Type>("a", 4).unwrap();
+ builder.append::<datatypes::Int64Type>("d", 11).unwrap();
+ let union = builder.build().unwrap();
+
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "union",
+ union.data_type().clone(),
+ false,
+ )]));
+
+ let union_array = Arc::new(union) as ArrayRef;
- let reader = ipc::reader::FileReader::try_new(std::io::Cursor::new(buf)).unwrap();
- let batch2: std::result::Result<Vec<_>, _> = reader.collect();
+ let rb = RecordBatch::try_new(schema, vec![union_array]).unwrap();
+ let rb2 = roundtrip_ipc(&rb);
+ // TODO: equality not yet implemented for union, so we check that the length of the array is
Review comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] paddyhoran commented on pull request #1140: feat(ipc): support for reading union arrays through IPC
Posted by GitBox <gi...@apache.org>.
paddyhoran commented on pull request #1140:
URL: https://github.com/apache/arrow-rs/pull/1140#issuecomment-1007431548
Thank you very much for this @helgikrs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org