[jira] [Updated] (ARROW-1011) [Format] Clarify requirements around buffer padding in validity bitmaps

["Rok Mihevc (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rok Mihevc updated ARROW-1011:
------------------------------
    External issue URL: https://github.com/apache/arrow/issues/16606

> [Format] Clarify requirements around buffer padding in validity bitmaps
> -----------------------------------------------------------------------
>
>                 Key: ARROW-1011
>                 URL: https://issues.apache.org/jira/browse/ARROW-1011
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Format
>            Reporter: Wes McKinney
>            Assignee: Wenchen Fan
>            Priority: Major
>             Fix For: 0.4.0
>
>
> This has come up in https://github.com/apache/arrow/pull/673 and in prior discussions. 
> The basic summary is that we should not write non-zero padding bytes in IPC messages. However: one cannot in general rely on the padding being non-zero when the data is in memory (for example: zero-copy slices of Arrow arrays/vectors).
> I think it would be good to clarify this point in Layout.md -- namely that what gets written to the wire should be deterministic. However, in-memory algorithms should not in general expect the padding region to have a particular value. As an example, a popcount on a validity bitmap would want to exclude padding bytes from the computation. Other elementwise SIMD operations are free to use the padding bytes as they wish, with a known caveat. 
> cc [~cloud_fan]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)