[jira] [Commented] (ARROW-11061) [Rust] Validate array properties against schema

["Neville Dipale (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-11061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255926#comment-17255926 ] 

Neville Dipale commented on ARROW-11061:
----------------------------------------

[~andygrove] [~alamb] [~jorgecarleitao] I don't know if you've encountered the issues above, but they're making my work veery difficult on the parquet writer. There are equivalent checks in the C++ implementation, but I haven't looked at them in detail yet.

> [Rust] Validate array properties against schema
> -----------------------------------------------
>
>                 Key: ARROW-11061
>                 URL: https://issues.apache.org/jira/browse/ARROW-11061
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Rust
>            Reporter: Neville Dipale
>            Priority: Major
>
> We have a problem when it comes to nested arrays, where one could create a <list[null]<primitive[non-null]>> where the array fields can't be null, but the list can have null slots.
> This creates a lot of work when working with such nested arrays, because we have to create work-arounds to account for this, and take unnecessarily slower paths.
> I propose that we prevent this problem at the source, by:
>  * checking that a batch can't be created with arrays that have incompatible null contracts
>  * preventing list and struct children from being non-null if any descendant of such children are null (might be less of an issue for structs)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)