You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/11/28 12:00:00 UTC
[jira] [Updated] (ARROW-9728) [Rust] [Parquet] Compute nested
spacing
[ https://issues.apache.org/jira/browse/ARROW-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated ARROW-9728:
----------------------------------
Labels: pull-request-available (was: )
> [Rust] [Parquet] Compute nested spacing
> ---------------------------------------
>
> Key: ARROW-9728
> URL: https://issues.apache.org/jira/browse/ARROW-9728
> Project: Apache Arrow
> Issue Type: Sub-task
> Components: Rust
> Affects Versions: 1.0.0
> Reporter: Neville Dipale
> Assignee: Neville Dipale
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When computing definition levels for deeply nested arrays that include lists, the definition levels are correctly calculated, but they are not translated into correct indexes for the eventual primitive arrays.
> For example, an int32 array could have no null values, but be a child of a list that has null values. If say the first 5 values of the int32 array are members of the first list item (i.e. list_array[0] = [1,2,3,4,5], and that list is itself a child of a struct whose index is null, the whole 5 values of the int32 array *should* be skipped. Further, the list's definition and repetition levels will be represented by 1 slot instead of the 5.
> The current logic cannot cater for this, and potentially results in slicing the int32 array incorrectly (sometimes including some of those first 5 values).
> This Jira is for the work necessary to compute the index into the eventual leaf arrays correctly.
> I started doing it as part of the initial writer PR, but it's complex and is blocking progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)