You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2023/09/29 15:26:19 UTC
[GitHub] [arrow-datafusion] sarutak commented on a diff in pull request #7663: fix: avro_to_arrow: Handle avro nested nullable struct (union)
sarutak commented on code in PR #7663:
URL: https://github.com/apache/arrow-datafusion/pull/7663#discussion_r1341503821
##########
datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs:
##########
@@ -1101,6 +1108,50 @@ mod test {
assert_eq!(batch.num_rows(), 3);
}
+ #[test]
+ fn test_avro_nullable_struct() {
+ let schema = apache_avro::Schema::parse_str(
+ r#"
+ {
+ "type": "record",
+ "name": "r1",
+ "fields": [
+ {
+ "name": "col1",
+ "type": ["null", {
+ "type": "record",
+ "name": "r2",
+ "fields": [
+ {
+ "name": "col2",
+ "type": "string"
+ }
+ ]
+ }],
+ "default": null
+ }
+ ]
+ }
+ "#,
+ )
+ .unwrap();
+ let mut r = apache_avro::types::Record::new(&schema).unwrap();
+ r.put("col1", AvroValue::Union(0, Box::new(AvroValue::Null)));
+
+ let mut w = apache_avro::Writer::new(&schema, vec![]);
+ w.append(r).unwrap();
+ let bytes = w.into_inner().unwrap();
+
+ let mut reader = ReaderBuilder::new()
+ .read_schema()
+ .with_batch_size(1)
+ .build(std::io::Cursor::new(bytes))
+ .unwrap();
+ let batch = reader.next().unwrap().unwrap();
+ assert_eq!(batch.num_rows(), 1);
+ assert_eq!(batch.num_columns(), 1);
Review Comment:
Should we test we can actually read the values written before.
##########
datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs:
##########
@@ -1101,6 +1108,50 @@ mod test {
assert_eq!(batch.num_rows(), 3);
}
+ #[test]
+ fn test_avro_nullable_struct() {
+ let schema = apache_avro::Schema::parse_str(
+ r#"
+ {
Review Comment:
Can we make indent more clean?
##########
datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs:
##########
@@ -1101,6 +1108,50 @@ mod test {
assert_eq!(batch.num_rows(), 3);
}
+ #[test]
+ fn test_avro_nullable_struct() {
+ let schema = apache_avro::Schema::parse_str(
+ r#"
+ {
+ "type": "record",
+ "name": "r1",
+ "fields": [
+ {
+ "name": "col1",
+ "type": ["null", {
+ "type": "record",
+ "name": "r2",
+ "fields": [
+ {
+ "name": "col2",
+ "type": "string"
+ }
+ ]
+ }],
+ "default": null
+ }
+ ]
+ }
+ "#,
+ )
+ .unwrap();
+ let mut r = apache_avro::types::Record::new(&schema).unwrap();
+ r.put("col1", AvroValue::Union(0, Box::new(AvroValue::Null)));
+
+ let mut w = apache_avro::Writer::new(&schema, vec![]);
+ w.append(r).unwrap();
Review Comment:
Is it better to add one more record which contains non-null `col1` to ensure it works even in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org