[jira] [Resolved] (ARROW-9728) [Rust] [Parquet] Compute nested definition and repetition for structs

["Andrew Lamb (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Lamb resolved ARROW-9728.
--------------------------------
    Fix Version/s: 3.0.0
       Resolution: Fixed

Issue resolved by pull request 8792
[https://github.com/apache/arrow/pull/8792]

> [Rust] [Parquet] Compute nested definition and repetition for structs
> ---------------------------------------------------------------------
>
>                 Key: ARROW-9728
>                 URL: https://issues.apache.org/jira/browse/ARROW-9728
>             Project: Apache Arrow
>          Issue Type: Sub-task
>          Components: Rust
>    Affects Versions: 1.0.0
>            Reporter: Neville Dipale
>            Assignee: Neville Dipale
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.0.0
>
>          Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> When computing definition levels for deeply nested arrays that include lists, the definition levels are correctly calculated, but they are not translated into correct indexes for the eventual primitive arrays.
> For example, an int32 array could have no null values, but be a child of a list that has null values. If say the first 5 values of the int32 array are members of the first list item (i.e. list_array[0] = [1,2,3,4,5], and that list is itself a child of a struct whose index is null, the whole 5 values of the int32 array *should* be skipped. Further, the list's definition and repetition levels will be represented by 1 slot instead of the 5.
> The current logic cannot cater for this, and potentially results in slicing the int32 array incorrectly (sometimes including some of those first 5 values).
> This Jira is for the work necessary to compute the index into the eventual leaf arrays correctly.
> I started doing it as part of the initial writer PR, but it's complex and is blocking progress.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)