You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Rok Mihevc (Jira)" <ji...@apache.org> on 2023/01/11 07:07:01 UTC

[jira] [Updated] (ARROW-39) C++: Logical chunked arrays / columns: conforming to fixed chunk sizes

     [ https://issues.apache.org/jira/browse/ARROW-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rok Mihevc updated ARROW-39:
----------------------------
    External issue URL: https://github.com/apache/arrow/issues/15413

> C++: Logical chunked arrays / columns: conforming to fixed chunk sizes
> ----------------------------------------------------------------------
>
>                 Key: ARROW-39
>                 URL: https://issues.apache.org/jira/browse/ARROW-39
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: C++
>            Reporter: Wes McKinney
>            Assignee: Wes McKinney
>            Priority: Major
>             Fix For: 0.3.0
>
>
> Implementing algorithms on large arrays assembled in physical chunks is problematic if:
> - The chunks are not all the same size (except possibly the last chunk, which can be less). Otherwise, retrieving a particular element is in general a O(log num_chunks) operation
> - The chunk size is not a power of 2. Computing integer modulus with a non-multiple of 2 requires more clock cycles (in other words, {{i % p}} is much more expensive to compute than {{i & (p - 1)}}, but the latter only works if p is a power of 2)
> Most of the Arrow data adapters will either feature contiguous data (1 chunk, so chunking is not an issue) or a regular chunk size, so this isn't as much of an immediate concern, but we should consider making it a contract of any data structures dealing in multiple arrays. 
> In general, it would be preferable to reorganize memory into either a regular chunksize (like 64K values per chunk) or a contiguous memory region. I would prefer for the moment to not to invest significant energy in writing algorithms for data with irregular chunk sizes. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)