[jira] [Resolved] (ARROW-16651) [Python] Casting Table to new schema ignores nullability of fields

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche resolved ARROW-16651.
-------------------------------------------
    Fix Version/s: 10.0.0
       Resolution: Fixed

Issue resolved by pull request 14048
[https://github.com/apache/arrow/pull/14048]

> [Python] Casting Table to new schema ignores nullability of fields
> ------------------------------------------------------------------
>
>                 Key: ARROW-16651
>                 URL: https://issues.apache.org/jira/browse/ARROW-16651
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Joris Van den Bossche
>            Assignee: Kshiteej K
>            Priority: Major
>              Labels: good-first-issue, good-second-issue, kernel, pull-request-available
>             Fix For: 10.0.0
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> Similar to ARROW-15478, but not for nested fields but just for casting a full Table (in theory that could be the same code, but currently the Table.cast logic is implemented in cython). 
> So currently when casting a Table to a new schema, the nullability of the fields in the schema is ignored (and as a result you get an "invalid" schema indicating a field is non-nullable that actually can have nulls):
> {code}
> >>> table = pa.table({'a': [None, 1]})
> >>> table
> pyarrow.Table
> a: int64
> ----
> a: [[null,1]]
> >>> new_schema = pa.schema([pa.field("a", "int64", nullable=False)])
> >>> table.cast(new_schema)
> pyarrow.Table
> a: int64 not null
> ----
> a: [[null,1]]
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)