You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/08/17 07:45:34 UTC
[GitHub] [arrow] Fokko opened a new issue, #37219: [Python] Nullability not maintained when casting a column
Fokko opened a new issue, #37219:
URL: https://github.com/apache/arrow/issues/37219
### Describe the bug, including details regarding any error messages, version, and platform.
Given the following Parquet file:
```
➜ arrow git:(arrow-14705) ✗ parquet-tools inspect ~/Desktop/int.parquet
############ file meta data ############
created_by: parquet-cpp-arrow version 12.0.0
num_columns: 1
num_rows: 3
num_row_groups: 1
format_version: 2.6
serialized_size: 798
############ Columns ############
id
############ Column(id) ############
name: id
path: id
max_definition_level: 0
max_repetition_level: 0
physical_type: INT32
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -5%)
```
When I widen a type, I see that the not-null is dropped from the schema:
I think this should be maintained.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on issue #37219: [Python] Nullability not maintained when casting a column
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on issue #37219:
URL: https://github.com/apache/arrow/issues/37219#issuecomment-1684105770
This works for bare projections of fields due to an explicit special case in [ProjectionDescr::FromExpressions](https://github.com/bkietz/arrow/blob/46cf7e67766f0646760acefa4d2d01cdfead2d5d/cpp/src/arrow/dataset/scanner.cc#L818-L823). It should be straightforward to extend that special case to also strip casts, so that `ds.field('id').cast(pa.int64())` inherits `id`'s nullability (and metadata).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Fokko commented on issue #37219: [Python] Nullability not maintained when casting a column
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #37219:
URL: https://github.com/apache/arrow/issues/37219#issuecomment-1688941267
Thanks @bkietz for the context and pointer. I was digging into the code, but was unable to see when the loop is actually executed. It seems that the size of `exprs` is always zero. I would except these tests to use the loop: https://github.com/apache/arrow/blob/b3936b9f0d0a899480741f2026c0349bf876c210/cpp/src/arrow/dataset/scanner_test.cc#L2060-L2062
Would you have a few more pointers where you would look to implement this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #37219: [Python] Nullability not maintained when casting a column
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37219:
URL: https://github.com/apache/arrow/issues/37219#issuecomment-1681805807
@bkietz @westonpace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Fokko commented on issue #37219: [Python] Nullability not maintained when casting a column
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #37219:
URL: https://github.com/apache/arrow/issues/37219#issuecomment-1681803473
cc @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on issue #37219: [Python] Nullability not maintained when casting a column
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on issue #37219:
URL: https://github.com/apache/arrow/issues/37219#issuecomment-1706655304
@Fokko I've tried debugging that unit test and I don't observe that `exprs.size() == 0` except for the first assertion
https://github.com/apache/arrow/blob/b3936b9f0d0a899480741f2026c0349bf876c210/cpp/src/arrow/dataset/scanner_test.cc#L2057
... where an empty projection is being tested. Is it possible that you were looking at that first assertion and never debugged through the subsequent one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org