You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jayshrivastava (via GitHub)" <gi...@apache.org> on 2023/04/13 21:38:03 UTC
[GitHub] [arrow] jayshrivastava opened a new issue, #35119: [Go] parquet: what is the correct way to differentiate an empty array and an array of size 1 containing NULL
jayshrivastava opened a new issue, #35119:
URL: https://github.com/apache/arrow/issues/35119
### Describe the usage question you have. Please include as many useful details as possible.
👋🏽 I have an schema field defined by `schema.NewPrimitiveNode("mycol", parquet.Repetitions.Repeated, parquet.Types.Int32, -1, -1)`. What's the correct way to encode an array which is empty?
I'm also not 100% sure about my understanding of def levels and rep levels. Here are some examples of data I'm encoding. For def levels, I put 1 when a value is defined and 0 when a value is null. For rep levels, I put a 0 for the first datum in an array and 1 for the remaining ones.
Values: [1, 2, 3], [4, 5]
Def Level: 1, 1, 1, 1, 1
Rep Level: 0, 1, 1, 0, 1
Values: [NULL, 5, 6]
Def Level: 0, 1, 1
Rep Level: 0, 1, 1
Values: [NULL], [1, 2, 3]
Def Level: 0, 1, 1, 1
Rep Level: 0, 0, 1, 1
Values: [NULL], [NULL]
Def Level: 0, 0
Rep Level: 0, 0
Values: [1], [], [2]
Def Level: 1, ?, 1
Rep Level: 0, ?, 0
I'm not sure how to represent the empty array since 0, 0 for (def, rep) represents an array of size 1 with a NULL value. If I don't put anything for the def level or rep level of the empty array, then a reader cannot tell that an empty array exists.
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jayshrivastava closed issue #35119: [Go] parquet: what is the correct way write repeated fields?
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava closed issue #35119: [Go] parquet: what is the correct way write repeated fields?
URL: https://github.com/apache/arrow/issues/35119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jayshrivastava commented on issue #35119: [Go] parquet: what is the correct way write repeated fields?
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on issue #35119:
URL: https://github.com/apache/arrow/issues/35119#issuecomment-1509988637
That was very helpful. Thanks!
Btw, I'm having some trouble inspecting the file with [parquet-mr on v1.13](https://formulae.brew.sh/formula/parquet-cli), but It looks like that's an issue with parquet-mr. I can inspect the file just fine on https://parquetreader.com/home or https://www.parquet-viewer.com/. This issue can be closed though.
[TestBasicDatums.zip](https://github.com/apache/arrow/files/11240782/TestBasicDatums.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #35119: [Go] parquet: what is the correct way write repeated fields?
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #35119:
URL: https://github.com/apache/arrow/issues/35119#issuecomment-1509273233
The misunderstanding is coming from the fact that you're using an old way to specify a list schema.
The schema you provided is:
```
Schema:
required group schema {
optional group a (LIST) {
repeated boolean a-datum;
}
optional group b (LIST) {
repeated boolean b-datum;
}
}
```
Note the usage of a 2-level format for describing the lists. From the Parquet Format spec, usage of this backwards-compatible format means that the elements are non-nullable:
```
// List<Integer> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated int32 element;
}
```
This means that the max definition level (describing whether data exists) is `2` not `3` like you assumed. If you try writing the values with a definition level of `2`, you should get the data you expect.
The *correct* way to specify a list would be as this:
```
repeated group field_id=-1 schema {
optional group field_id=-1 a (List) {
repeated group field_id=-1 list {
optional boolean field_id=-1 element;
}
}
optional group field_id=-1 b (List) {
repeated group field_id=-1 list {
optional boolean field_id=-1 element;
}
}
}
```
Note the 3-level format here. This enables specifying that the elements can be null and will result in a maximum definition level of `3` like you were expecting, where a `2` would indicate a null in the list.
Hope this helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org