You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jayshrivastava (via GitHub)" <gi...@apache.org> on 2023/05/16 15:54:09 UTC
[GitHub] [arrow] jayshrivastava opened a new issue, #35619: [Go] parquet: is there a way to differentiate null tuples and tuples containing null?
jayshrivastava opened a new issue, #35619:
URL: https://github.com/apache/arrow/issues/35619
### Describe the usage question you have. Please include as many useful details as possible.
Using the following schema to represent a nullable integer tuple type, how does one write the tuple `NULL` and differentiate it from `(NULL, NULL)`? I know the latter can be written by writing integers with def level 0, but I'm not sure about the former.
```
message schema {
optional group a (LIST) {
optional int f1;
optional int f2;
}
}
```
If it's not possible, are there any methods (ex. recommended schemas, columns, metadata) which are commonly used in this situation? I was thinking about adding an additional bool field `isNull` to deal with the problem, but I'm not sure if it's the best way.
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1549958652
@jayshrivastava in addition to what @mapleFU stated, this is also why the standard way for creating a list of tuples uses a 3-level schema that looks like this:
```
required group schema {
optional group a (LIST) {
repeated group tuples {
optional int f1;
optional int f2;
}
}
}
```
rather than the 2-level version in your issue. This can be seen via the description on the parquet format document: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists
The fact that readers can interpret your original version is because the spec states that it should be supported for backwards compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jayshrivastava commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1549954017
@mapleFU What def levels and rep levels should I use in each case? My understanding is that rep levels are not relevant since there are no repeated fields. Can you please clarify what you mean by parquet internal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1549959017
I guess it could be handled by Go library, just using arrow to write is ok.
For understanding parquet internal, the blogs might help:
1. https://blog.twitter.com/engineering/en_us/a/2013/dremel-made-simple-with-parquet
2. https://arrow.apache.org/blog/2022/10/08/arrow-parquet-encoding-part-2/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jayshrivastava commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1550097700
@zeroshade @mapleFU Thanks for the help. I appreciate it a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1549948590
@jayshrivastava Hi jay, this could handle by parquet internal. Parquet format has rep-level and def-level. `null` and `(null, null)` would have different rep-level and def-levels. So, Parquet reader and writer can distinct them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #35619:
URL: https://github.com/apache/arrow/issues/35619#issuecomment-1549971319
> @mapleFU What def levels and rep levels should I use in each case? My understanding is that rep levels are not relevant since there are no repeated fields. Can you please clarify what you mean by parquet internal?
You're correct here. Repetition levels aren't relevant in this case, only definition levels. Using your original schema:
```
message schema {
optional group a (LIST) {
optional int f1;
optional int f2;
}
}
```
You've got two columns: `f1` and `f2`.
If both are non-null, then their definition levels would be `2` to indicate a value. To indicate `a` is null (ie: the whole tuple is `null`) you would use the definition level of 0 for both. To indicate the tuple is not-null but the values are (ie: your `(null, null)` case) then you use the definition level of 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jayshrivastava closed issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava closed issue #35619: [Go] parquet: is there a way to differentiate null groups and groups containing null?
URL: https://github.com/apache/arrow/issues/35619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org