[jira] [Created] (ARROW-15687) [Format] Clarify that 8 byte padding must not be applied to compressed buffers

["Jorge Leitão (Jira)" <ji...@apache.org>]

Jorge Leitão created ARROW-15687:
------------------------------------

             Summary: [Format] Clarify that 8 byte padding must not be applied to compressed buffers
                 Key: ARROW-15687
                 URL: https://issues.apache.org/jira/browse/ARROW-15687
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Format
            Reporter: Jorge Leitão


I was unable to find where this is discussed, but I think we do not mention that 8 byte padding must not be applied when the buffer is compressed, as it causes us to lose the size of the compressed buffer.

For example

```
import pyarrow.ipc

data = [
    pyarrow.array([1, 2, 3, 4, 5], type="int32"),
]

batch = pyarrow.record_batch(data, names=['f0'])

with pyarrow.OSFile('test1.arrow', 'wb') as sink:
    with pyarrow.ipc.new_file(sink, batch.schema, options=pyarrow.ipc.IpcWriteOptions(compression="zstd")) as writer:
        writer.write(batch)
```

outputs a single data buffer with

```
[20, 0, 0, 0, 0, 0, 0, 0, 40, 181, 47, 253, 32, 20, 161, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0, 0, 0]
```
which has 37 bytes (padding would require 40 bytes).

My understanding is that we do not pad because doing so make us unable to recover the original size of the (compressed) data, and offers no advantage since users can't mmap data anyways.




--
This message was sent by Atlassian Jira
(v8.20.1#820001)