You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Jacob Quinn <qu...@gmail.com> on 2021/04/07 05:39:37 UTC
Alignment not stored in arrow metadata
As far as I can tell, the alignment padding used in an IPC stream/file
isn't stored explicitly, and not really "inferrable", though maybe
technically possible if you calculated what bytes are *necessary* given a
buffer's data vs. what's actually stored.
Just wondering if this has been brought up at all to store explicitly; it
came up in the Julia implementation when considering "appending" record
batches to an IPC stream that has already been written to disk; we
originally thought we would need to match alignment used in previously
written record batches, but upon further reflection, it seems like
technically it wouldn't matter since all buffers have the exact byte counts
written anyway. Just wasn't sure if it would be breaking implicit
assumptions by consumers somewhere if they happened to get an IPC stream w/
record batches that mixed, for example, 8-byte and 64-byte alignments.
-Jacob
Re: Alignment not stored in arrow metadata
Posted by Micah Kornfield <em...@gmail.com>.
>
> AFAIK IPC is just bytes. The alignment is done when they are copied over to
> allocated memory regions.
Agreed, that if implementations are copying then this isn't a concern. The
IPC and File Formats were designed for memory mapping/zero copy. So there
is an assumption that kernel pages meet the alignment requirements but
otherwise a copy should not be strictly necessary.
Just wasn't sure if it would be breaking implicit
> assumptions by consumers somewhere if they happened to get an IPC stream w/
> record batches that mixed, for example, 8-byte and 64-byte alignments.
I'm not aware of any assumptions here. Simply given the fact that there
isn't a mandate (and due to things like slicing, sharing buffers via the
C-ABI etc), I think all code handling Arrow arrays that wants to optimize
for an alignment still needs to verify alignment requirements on a buffer
by buffer basis.
On Wed, Apr 7, 2021 at 3:23 AM Jorge Cardoso Leitão <
jorgecarleitao@gmail.com> wrote:
> Hi Jacob,
>
> AFAIK IPC is just bytes. The alignment is done when they are copied over to
> allocated memory regions. It is the implementations' responsibility to
> allocate memory regions that are aligned depending on how those bytes
> should be interpreted (e.g. u64 vs u8). This interpretation is induced by
> the relationship between the logical types (e.g. Time32) and its
> corresponding physical types (e.g. 0th buffer is u8, 1st is i32). In this
> sense, afaik IPC does not need to declare byte alignment as they are
> inferred by the corresponding logical type.
>
> Best,
> Jorge
>
>
>
>
> On Wed, Apr 7, 2021 at 7:40 AM Jacob Quinn <qu...@gmail.com> wrote:
>
> > As far as I can tell, the alignment padding used in an IPC stream/file
> > isn't stored explicitly, and not really "inferrable", though maybe
> > technically possible if you calculated what bytes are *necessary* given a
> > buffer's data vs. what's actually stored.
> >
> > Just wondering if this has been brought up at all to store explicitly; it
> > came up in the Julia implementation when considering "appending" record
> > batches to an IPC stream that has already been written to disk; we
> > originally thought we would need to match alignment used in previously
> > written record batches, but upon further reflection, it seems like
> > technically it wouldn't matter since all buffers have the exact byte
> counts
> > written anyway. Just wasn't sure if it would be breaking implicit
> > assumptions by consumers somewhere if they happened to get an IPC stream
> w/
> > record batches that mixed, for example, 8-byte and 64-byte alignments.
> >
> > -Jacob
> >
>
Re: Alignment not stored in arrow metadata
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Hi Jacob,
AFAIK IPC is just bytes. The alignment is done when they are copied over to
allocated memory regions. It is the implementations' responsibility to
allocate memory regions that are aligned depending on how those bytes
should be interpreted (e.g. u64 vs u8). This interpretation is induced by
the relationship between the logical types (e.g. Time32) and its
corresponding physical types (e.g. 0th buffer is u8, 1st is i32). In this
sense, afaik IPC does not need to declare byte alignment as they are
inferred by the corresponding logical type.
Best,
Jorge
On Wed, Apr 7, 2021 at 7:40 AM Jacob Quinn <qu...@gmail.com> wrote:
> As far as I can tell, the alignment padding used in an IPC stream/file
> isn't stored explicitly, and not really "inferrable", though maybe
> technically possible if you calculated what bytes are *necessary* given a
> buffer's data vs. what's actually stored.
>
> Just wondering if this has been brought up at all to store explicitly; it
> came up in the Julia implementation when considering "appending" record
> batches to an IPC stream that has already been written to disk; we
> originally thought we would need to match alignment used in previously
> written record batches, but upon further reflection, it seems like
> technically it wouldn't matter since all buffers have the exact byte counts
> written anyway. Just wasn't sure if it would be breaking implicit
> assumptions by consumers somewhere if they happened to get an IPC stream w/
> record batches that mixed, for example, 8-byte and 64-byte alignments.
>
> -Jacob
>