You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Emilio Lahr-Vivaz <el...@ccri.com> on 2017/08/08 12:08:28 UTC
buffer alignment (format/java/js)
I'm looking into buffer alignment in the java writer classes. Currently
some files written with the java streaming writer can't be read due to
the javascript TypedArray's restriction that the start offset of the
array must be a multiple of the data size of the array type (i.e.
Int32Vectors must start on a multiple of 4, Float64Vectors must start on
a multiple of 8, etc). From a cursory look at the java writer, I believe
that the schema that is written first is not aligned at all, and then
each record batch pads out its size to a multiple of 8. So:
1. should the schema block pad itself so that the first record batch is
aligned, and is there any problem with doing so?
2. is there any problem with changing the alignment to 64 bytes, as
recommended (but not required) by the spec?
Thanks,
Emilio
Re: buffer alignment (format/java/js)
Posted by Wes McKinney <we...@gmail.com>.
Thanks Emilio. I left some more comments on the PR. I'm going to make
a pass through the C++ implementation and add some paranoia checks on
both the write path and the read path. I'll push these changes to your
PR branch so we have things integration tested together.
On the write path, per ARROW-1224 I think we can relax buffer padding
to 8 byte multiples rather than 64 bytes. I will also make sure the
schema message is being properly padded.
On the read path, I'll add some debug assertions that all buffers
start on aligned offsets, and that message bodies are always a
multiple of 8. This will help enforce these contracts with other Arrow
implementations when integration testing.
- Wes
On Tue, Aug 8, 2017 at 2:11 PM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
> I opened a PR for comments here: https://github.com/apache/arrow/pull/954
>
> So far I've just added some padding to the end of the buffers. I probably
> will need to pad the schema out as well... I'm testing things out against
> the js code at the moment.
>
>
> On 08/08/2017 02:06 PM, Wes McKinney wrote:
>>
>> I opened https://issues.apache.org/jira/browse/ARROW-1343. Let's try
>> to resolve this today so it can make the 0.6.0 release?
>>
>> On Tue, Aug 8, 2017 at 2:03 PM, Wes McKinney <we...@gmail.com> wrote:
>>>
>>> I'm happy to have a look at the branch / integration tests if you
>>> could put up a PR
>>>
>>> When you say "a single serialized record batch" you mean an
>>> encapsulated message, right (including length prefix and metadata)?
>>> Using the terminology from http://arrow.apache.org/docs/ipc.html. I
>>> guess the problem is that the total size of the Schema message at the
>>> start of the stream may not be a multiple of 8. We should totally fix
>>> this; I don't think it even constitutes a breakage of the format -- I
>>> am fairly sure with extra padding bytes between the schema and the
>>> first record batch (or first dictionary) that the stream will be
>>> backwards compatible.
>>>
>>> However, we should document in
>>> https://github.com/apache/arrow/blob/master/format/IPC.md that message
>>> sizes are expected to be a multiple of 8. We should also take a look
>>> at the File format implementation to ensure that padding is inserted
>>> after the magic number at the start of the file
>>>
>>> - Wes
>>>
>>> On Tue, Aug 8, 2017 at 1:32 PM, Emilio Lahr-Vivaz <el...@ccri.com>
>>> wrote:
>>>>
>>>> Sure, the workflow is a little complicated, but we have the following
>>>> code
>>>> running in distributed databases (as accumulo iterators and hbase
>>>> coprocessors). They process data rows and transform them into arrow
>>>> records,
>>>> then periodically write out record batches:
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L79-L105
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/master/geomesa-arrow/geomesa-arrow-gt/src/main/scala/org/locationtech/geomesa/arrow/io/records/RecordBatchUnloader.scala#L20
>>>>
>>>> The record batches come back to a single client and are concatenated
>>>> with a
>>>> file header and footer (then wrapped in SimpleFeature objects, as we
>>>> implement a geotools data store):
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L265-L268
>>>>
>>>> The resulting bytes are written out as an arrow streaming file that we
>>>> parse
>>>> with the arrow-js libraries in the browser.
>>>>
>>>> Thanks,
>>>>
>>>> Emilio
>>>>
>>>>
>>>> On 08/08/2017 01:24 PM, Li Jin wrote:
>>>>>
>>>>> Hi Emilio,
>>>>>
>>>>>> So I think the issue is that we are serializing record batches in a
>>>>>
>>>>> distributed fashion, and then > concatenating them in the streaming
>>>>> format.
>>>>>
>>>>> Can you show the code for this?
>>>>>
>>>>> On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz
>>>>> <el...@ccri.com>
>>>>> wrote:
>>>>>
>>>>>> So I think the issue is that we are serializing record batches in a
>>>>>> distributed fashion, and then concatenating them in the streaming
>>>>>> format.
>>>>>> However, the message serialization only aligns the start of the
>>>>>> buffers,
>>>>>> which requires it to know the current absolute offset of the output
>>>>>> stream.
>>>>>> Would there be any problem with padding the end of the message, so any
>>>>>> single serialized record batch would always be a multiple of 8 bytes?
>>>>>>
>>>>>> I've put together a branch that does this, and the existing java tests
>>>>>> all
>>>>>> pass. I'm having some trouble running the integration tests though.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Emilio
>>>>>>
>>>>>>
>>>>>> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>>>>>>
>>>>>>> Hi Wes,
>>>>>>>
>>>>>>> You're right, I just realized that. I think the alignment issue might
>>>>>>> be
>>>>>>> in some unrelated code, actually. From what I can tell the the arrow
>>>>>>> writers are aligning buffers correctly; if not I'll open a bug.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Emilio
>>>>>>>
>>>>>>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>>>>>>
>>>>>>>> hi Emilio,
>>>>>>>>
>>>>>>>> From your description, it isn't clear why 8-byte alignment is
>>>>>>>> causing
>>>>>>>> a problem (as compare with 64-byte alignment). My understanding is
>>>>>>>> that JavaScript's TypedArray classes range in size from 1 to 8
>>>>>>>> bytes.
>>>>>>>>
>>>>>>>> The starting offset for all buffers should be 8-byte aligned, if not
>>>>>>>> that is a bug. Could you clarify?
>>>>>>>>
>>>>>>>> - Wes
>>>>>>>>
>>>>>>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz
>>>>>>>> <el...@ccri.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> After looking at it further, I think only the buffers themselves
>>>>>>>>> need
>>>>>>>>> to be
>>>>>>>>> aligned, not the metadata and/or schema. Would there be any problem
>>>>>>>>> with
>>>>>>>>> changing the alignment to 64 bytes then?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Emilio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>>>>>>
>>>>>>>>>> I'm looking into buffer alignment in the java writer classes.
>>>>>>>>>> Currently
>>>>>>>>>> some files written with the java streaming writer can't be read
>>>>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>> javascript TypedArray's restriction that the start offset of the
>>>>>>>>>> array
>>>>>>>>>> must
>>>>>>>>>> be a multiple of the data size of the array type (i.e.
>>>>>>>>>> Int32Vectors
>>>>>>>>>> must
>>>>>>>>>> start on a multiple of 4, Float64Vectors must start on a multiple
>>>>>>>>>> of
>>>>>>>>>> 8,
>>>>>>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>>>>>>> schema that
>>>>>>>>>> is written first is not aligned at all, and then each record batch
>>>>>>>>>> pads out
>>>>>>>>>> its size to a multiple of 8. So:
>>>>>>>>>>
>>>>>>>>>> 1. should the schema block pad itself so that the first record
>>>>>>>>>> batch
>>>>>>>>>> is
>>>>>>>>>> aligned, and is there any problem with doing so?
>>>>>>>>>> 2. is there any problem with changing the alignment to 64 bytes,
>>>>>>>>>> as
>>>>>>>>>> recommended (but not required) by the spec?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Emilio
>>>>>>>>>>
>
Re: buffer alignment (format/java/js)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
I opened a PR for comments here: https://github.com/apache/arrow/pull/954
So far I've just added some padding to the end of the buffers. I
probably will need to pad the schema out as well... I'm testing things
out against the js code at the moment.
On 08/08/2017 02:06 PM, Wes McKinney wrote:
> I opened https://issues.apache.org/jira/browse/ARROW-1343. Let's try
> to resolve this today so it can make the 0.6.0 release?
>
> On Tue, Aug 8, 2017 at 2:03 PM, Wes McKinney <we...@gmail.com> wrote:
>> I'm happy to have a look at the branch / integration tests if you
>> could put up a PR
>>
>> When you say "a single serialized record batch" you mean an
>> encapsulated message, right (including length prefix and metadata)?
>> Using the terminology from http://arrow.apache.org/docs/ipc.html. I
>> guess the problem is that the total size of the Schema message at the
>> start of the stream may not be a multiple of 8. We should totally fix
>> this; I don't think it even constitutes a breakage of the format -- I
>> am fairly sure with extra padding bytes between the schema and the
>> first record batch (or first dictionary) that the stream will be
>> backwards compatible.
>>
>> However, we should document in
>> https://github.com/apache/arrow/blob/master/format/IPC.md that message
>> sizes are expected to be a multiple of 8. We should also take a look
>> at the File format implementation to ensure that padding is inserted
>> after the magic number at the start of the file
>>
>> - Wes
>>
>> On Tue, Aug 8, 2017 at 1:32 PM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
>>> Sure, the workflow is a little complicated, but we have the following code
>>> running in distributed databases (as accumulo iterators and hbase
>>> coprocessors). They process data rows and transform them into arrow records,
>>> then periodically write out record batches:
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L79-L105
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-arrow/geomesa-arrow-gt/src/main/scala/org/locationtech/geomesa/arrow/io/records/RecordBatchUnloader.scala#L20
>>>
>>> The record batches come back to a single client and are concatenated with a
>>> file header and footer (then wrapped in SimpleFeature objects, as we
>>> implement a geotools data store):
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L265-L268
>>>
>>> The resulting bytes are written out as an arrow streaming file that we parse
>>> with the arrow-js libraries in the browser.
>>>
>>> Thanks,
>>>
>>> Emilio
>>>
>>>
>>> On 08/08/2017 01:24 PM, Li Jin wrote:
>>>> Hi Emilio,
>>>>
>>>>> So I think the issue is that we are serializing record batches in a
>>>> distributed fashion, and then > concatenating them in the streaming
>>>> format.
>>>>
>>>> Can you show the code for this?
>>>>
>>>> On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz <el...@ccri.com>
>>>> wrote:
>>>>
>>>>> So I think the issue is that we are serializing record batches in a
>>>>> distributed fashion, and then concatenating them in the streaming format.
>>>>> However, the message serialization only aligns the start of the buffers,
>>>>> which requires it to know the current absolute offset of the output
>>>>> stream.
>>>>> Would there be any problem with padding the end of the message, so any
>>>>> single serialized record batch would always be a multiple of 8 bytes?
>>>>>
>>>>> I've put together a branch that does this, and the existing java tests
>>>>> all
>>>>> pass. I'm having some trouble running the integration tests though.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Emilio
>>>>>
>>>>>
>>>>> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>>>>>
>>>>>> Hi Wes,
>>>>>>
>>>>>> You're right, I just realized that. I think the alignment issue might be
>>>>>> in some unrelated code, actually. From what I can tell the the arrow
>>>>>> writers are aligning buffers correctly; if not I'll open a bug.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Emilio
>>>>>>
>>>>>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>>>>>
>>>>>>> hi Emilio,
>>>>>>>
>>>>>>> From your description, it isn't clear why 8-byte alignment is causing
>>>>>>> a problem (as compare with 64-byte alignment). My understanding is
>>>>>>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>>>>>>
>>>>>>> The starting offset for all buffers should be 8-byte aligned, if not
>>>>>>> that is a bug. Could you clarify?
>>>>>>>
>>>>>>> - Wes
>>>>>>>
>>>>>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> After looking at it further, I think only the buffers themselves need
>>>>>>>> to be
>>>>>>>> aligned, not the metadata and/or schema. Would there be any problem
>>>>>>>> with
>>>>>>>> changing the alignment to 64 bytes then?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Emilio
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>>>>>
>>>>>>>>> I'm looking into buffer alignment in the java writer classes.
>>>>>>>>> Currently
>>>>>>>>> some files written with the java streaming writer can't be read due
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> javascript TypedArray's restriction that the start offset of the
>>>>>>>>> array
>>>>>>>>> must
>>>>>>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>>>>>>> must
>>>>>>>>> start on a multiple of 4, Float64Vectors must start on a multiple of
>>>>>>>>> 8,
>>>>>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>>>>>> schema that
>>>>>>>>> is written first is not aligned at all, and then each record batch
>>>>>>>>> pads out
>>>>>>>>> its size to a multiple of 8. So:
>>>>>>>>>
>>>>>>>>> 1. should the schema block pad itself so that the first record batch
>>>>>>>>> is
>>>>>>>>> aligned, and is there any problem with doing so?
>>>>>>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>>>>>>> recommended (but not required) by the spec?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Emilio
>>>>>>>>>
Re: buffer alignment (format/java/js)
Posted by Wes McKinney <we...@gmail.com>.
I opened https://issues.apache.org/jira/browse/ARROW-1343. Let's try
to resolve this today so it can make the 0.6.0 release?
On Tue, Aug 8, 2017 at 2:03 PM, Wes McKinney <we...@gmail.com> wrote:
> I'm happy to have a look at the branch / integration tests if you
> could put up a PR
>
> When you say "a single serialized record batch" you mean an
> encapsulated message, right (including length prefix and metadata)?
> Using the terminology from http://arrow.apache.org/docs/ipc.html. I
> guess the problem is that the total size of the Schema message at the
> start of the stream may not be a multiple of 8. We should totally fix
> this; I don't think it even constitutes a breakage of the format -- I
> am fairly sure with extra padding bytes between the schema and the
> first record batch (or first dictionary) that the stream will be
> backwards compatible.
>
> However, we should document in
> https://github.com/apache/arrow/blob/master/format/IPC.md that message
> sizes are expected to be a multiple of 8. We should also take a look
> at the File format implementation to ensure that padding is inserted
> after the magic number at the start of the file
>
> - Wes
>
> On Tue, Aug 8, 2017 at 1:32 PM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
>> Sure, the workflow is a little complicated, but we have the following code
>> running in distributed databases (as accumulo iterators and hbase
>> coprocessors). They process data rows and transform them into arrow records,
>> then periodically write out record batches:
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L79-L105
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-arrow/geomesa-arrow-gt/src/main/scala/org/locationtech/geomesa/arrow/io/records/RecordBatchUnloader.scala#L20
>>
>> The record batches come back to a single client and are concatenated with a
>> file header and footer (then wrapped in SimpleFeature objects, as we
>> implement a geotools data store):
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L265-L268
>>
>> The resulting bytes are written out as an arrow streaming file that we parse
>> with the arrow-js libraries in the browser.
>>
>> Thanks,
>>
>> Emilio
>>
>>
>> On 08/08/2017 01:24 PM, Li Jin wrote:
>>>
>>> Hi Emilio,
>>>
>>>> So I think the issue is that we are serializing record batches in a
>>>
>>> distributed fashion, and then > concatenating them in the streaming
>>> format.
>>>
>>> Can you show the code for this?
>>>
>>> On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz <el...@ccri.com>
>>> wrote:
>>>
>>>> So I think the issue is that we are serializing record batches in a
>>>> distributed fashion, and then concatenating them in the streaming format.
>>>> However, the message serialization only aligns the start of the buffers,
>>>> which requires it to know the current absolute offset of the output
>>>> stream.
>>>> Would there be any problem with padding the end of the message, so any
>>>> single serialized record batch would always be a multiple of 8 bytes?
>>>>
>>>> I've put together a branch that does this, and the existing java tests
>>>> all
>>>> pass. I'm having some trouble running the integration tests though.
>>>>
>>>> Thanks,
>>>>
>>>> Emilio
>>>>
>>>>
>>>> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>>>>
>>>>> Hi Wes,
>>>>>
>>>>> You're right, I just realized that. I think the alignment issue might be
>>>>> in some unrelated code, actually. From what I can tell the the arrow
>>>>> writers are aligning buffers correctly; if not I'll open a bug.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Emilio
>>>>>
>>>>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>>>>
>>>>>> hi Emilio,
>>>>>>
>>>>>> From your description, it isn't clear why 8-byte alignment is causing
>>>>>> a problem (as compare with 64-byte alignment). My understanding is
>>>>>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>>>>>
>>>>>> The starting offset for all buffers should be 8-byte aligned, if not
>>>>>> that is a bug. Could you clarify?
>>>>>>
>>>>>> - Wes
>>>>>>
>>>>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com>
>>>>>> wrote:
>>>>>>
>>>>>>> After looking at it further, I think only the buffers themselves need
>>>>>>> to be
>>>>>>> aligned, not the metadata and/or schema. Would there be any problem
>>>>>>> with
>>>>>>> changing the alignment to 64 bytes then?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Emilio
>>>>>>>
>>>>>>>
>>>>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>>>>
>>>>>>>> I'm looking into buffer alignment in the java writer classes.
>>>>>>>> Currently
>>>>>>>> some files written with the java streaming writer can't be read due
>>>>>>>> to
>>>>>>>> the
>>>>>>>> javascript TypedArray's restriction that the start offset of the
>>>>>>>> array
>>>>>>>> must
>>>>>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>>>>>> must
>>>>>>>> start on a multiple of 4, Float64Vectors must start on a multiple of
>>>>>>>> 8,
>>>>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>>>>> schema that
>>>>>>>> is written first is not aligned at all, and then each record batch
>>>>>>>> pads out
>>>>>>>> its size to a multiple of 8. So:
>>>>>>>>
>>>>>>>> 1. should the schema block pad itself so that the first record batch
>>>>>>>> is
>>>>>>>> aligned, and is there any problem with doing so?
>>>>>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>>>>>> recommended (but not required) by the spec?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Emilio
>>>>>>>>
>>>>>>>
>>
Re: buffer alignment (format/java/js)
Posted by Wes McKinney <we...@gmail.com>.
I'm happy to have a look at the branch / integration tests if you
could put up a PR
When you say "a single serialized record batch" you mean an
encapsulated message, right (including length prefix and metadata)?
Using the terminology from http://arrow.apache.org/docs/ipc.html. I
guess the problem is that the total size of the Schema message at the
start of the stream may not be a multiple of 8. We should totally fix
this; I don't think it even constitutes a breakage of the format -- I
am fairly sure with extra padding bytes between the schema and the
first record batch (or first dictionary) that the stream will be
backwards compatible.
However, we should document in
https://github.com/apache/arrow/blob/master/format/IPC.md that message
sizes are expected to be a multiple of 8. We should also take a look
at the File format implementation to ensure that padding is inserted
after the magic number at the start of the file
- Wes
On Tue, Aug 8, 2017 at 1:32 PM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
> Sure, the workflow is a little complicated, but we have the following code
> running in distributed databases (as accumulo iterators and hbase
> coprocessors). They process data rows and transform them into arrow records,
> then periodically write out record batches:
>
> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L79-L105
>
> https://github.com/locationtech/geomesa/blob/master/geomesa-arrow/geomesa-arrow-gt/src/main/scala/org/locationtech/geomesa/arrow/io/records/RecordBatchUnloader.scala#L20
>
> The record batches come back to a single client and are concatenated with a
> file header and footer (then wrapped in SimpleFeature objects, as we
> implement a geotools data store):
>
> https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L265-L268
>
> The resulting bytes are written out as an arrow streaming file that we parse
> with the arrow-js libraries in the browser.
>
> Thanks,
>
> Emilio
>
>
> On 08/08/2017 01:24 PM, Li Jin wrote:
>>
>> Hi Emilio,
>>
>>> So I think the issue is that we are serializing record batches in a
>>
>> distributed fashion, and then > concatenating them in the streaming
>> format.
>>
>> Can you show the code for this?
>>
>> On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz <el...@ccri.com>
>> wrote:
>>
>>> So I think the issue is that we are serializing record batches in a
>>> distributed fashion, and then concatenating them in the streaming format.
>>> However, the message serialization only aligns the start of the buffers,
>>> which requires it to know the current absolute offset of the output
>>> stream.
>>> Would there be any problem with padding the end of the message, so any
>>> single serialized record batch would always be a multiple of 8 bytes?
>>>
>>> I've put together a branch that does this, and the existing java tests
>>> all
>>> pass. I'm having some trouble running the integration tests though.
>>>
>>> Thanks,
>>>
>>> Emilio
>>>
>>>
>>> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>>>
>>>> Hi Wes,
>>>>
>>>> You're right, I just realized that. I think the alignment issue might be
>>>> in some unrelated code, actually. From what I can tell the the arrow
>>>> writers are aligning buffers correctly; if not I'll open a bug.
>>>>
>>>> Thanks,
>>>>
>>>> Emilio
>>>>
>>>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>>>
>>>>> hi Emilio,
>>>>>
>>>>> From your description, it isn't clear why 8-byte alignment is causing
>>>>> a problem (as compare with 64-byte alignment). My understanding is
>>>>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>>>>
>>>>> The starting offset for all buffers should be 8-byte aligned, if not
>>>>> that is a bug. Could you clarify?
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com>
>>>>> wrote:
>>>>>
>>>>>> After looking at it further, I think only the buffers themselves need
>>>>>> to be
>>>>>> aligned, not the metadata and/or schema. Would there be any problem
>>>>>> with
>>>>>> changing the alignment to 64 bytes then?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Emilio
>>>>>>
>>>>>>
>>>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>>>
>>>>>>> I'm looking into buffer alignment in the java writer classes.
>>>>>>> Currently
>>>>>>> some files written with the java streaming writer can't be read due
>>>>>>> to
>>>>>>> the
>>>>>>> javascript TypedArray's restriction that the start offset of the
>>>>>>> array
>>>>>>> must
>>>>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>>>>> must
>>>>>>> start on a multiple of 4, Float64Vectors must start on a multiple of
>>>>>>> 8,
>>>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>>>> schema that
>>>>>>> is written first is not aligned at all, and then each record batch
>>>>>>> pads out
>>>>>>> its size to a multiple of 8. So:
>>>>>>>
>>>>>>> 1. should the schema block pad itself so that the first record batch
>>>>>>> is
>>>>>>> aligned, and is there any problem with doing so?
>>>>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>>>>> recommended (but not required) by the spec?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Emilio
>>>>>>>
>>>>>>
>
Re: buffer alignment (format/java/js)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
Sure, the workflow is a little complicated, but we have the following
code running in distributed databases (as accumulo iterators and hbase
coprocessors). They process data rows and transform them into arrow
records, then periodically write out record batches:
https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L79-L105
https://github.com/locationtech/geomesa/blob/master/geomesa-arrow/geomesa-arrow-gt/src/main/scala/org/locationtech/geomesa/arrow/io/records/RecordBatchUnloader.scala#L20
The record batches come back to a single client and are concatenated
with a file header and footer (then wrapped in SimpleFeature objects, as
we implement a geotools data store):
https://github.com/locationtech/geomesa/blob/master/geomesa-index-api/src/main/scala/org/locationtech/geomesa/index/iterators/ArrowBatchScan.scala#L265-L268
The resulting bytes are written out as an arrow streaming file that we
parse with the arrow-js libraries in the browser.
Thanks,
Emilio
On 08/08/2017 01:24 PM, Li Jin wrote:
> Hi Emilio,
>
>> So I think the issue is that we are serializing record batches in a
> distributed fashion, and then > concatenating them in the streaming format.
>
> Can you show the code for this?
>
> On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz <el...@ccri.com>
> wrote:
>
>> So I think the issue is that we are serializing record batches in a
>> distributed fashion, and then concatenating them in the streaming format.
>> However, the message serialization only aligns the start of the buffers,
>> which requires it to know the current absolute offset of the output stream.
>> Would there be any problem with padding the end of the message, so any
>> single serialized record batch would always be a multiple of 8 bytes?
>>
>> I've put together a branch that does this, and the existing java tests all
>> pass. I'm having some trouble running the integration tests though.
>>
>> Thanks,
>>
>> Emilio
>>
>>
>> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>>
>>> Hi Wes,
>>>
>>> You're right, I just realized that. I think the alignment issue might be
>>> in some unrelated code, actually. From what I can tell the the arrow
>>> writers are aligning buffers correctly; if not I'll open a bug.
>>>
>>> Thanks,
>>>
>>> Emilio
>>>
>>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>>
>>>> hi Emilio,
>>>>
>>>> From your description, it isn't clear why 8-byte alignment is causing
>>>> a problem (as compare with 64-byte alignment). My understanding is
>>>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>>>
>>>> The starting offset for all buffers should be 8-byte aligned, if not
>>>> that is a bug. Could you clarify?
>>>>
>>>> - Wes
>>>>
>>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com>
>>>> wrote:
>>>>
>>>>> After looking at it further, I think only the buffers themselves need
>>>>> to be
>>>>> aligned, not the metadata and/or schema. Would there be any problem with
>>>>> changing the alignment to 64 bytes then?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Emilio
>>>>>
>>>>>
>>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>>
>>>>>> I'm looking into buffer alignment in the java writer classes. Currently
>>>>>> some files written with the java streaming writer can't be read due to
>>>>>> the
>>>>>> javascript TypedArray's restriction that the start offset of the array
>>>>>> must
>>>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>>>> must
>>>>>> start on a multiple of 4, Float64Vectors must start on a multiple of 8,
>>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>>> schema that
>>>>>> is written first is not aligned at all, and then each record batch
>>>>>> pads out
>>>>>> its size to a multiple of 8. So:
>>>>>>
>>>>>> 1. should the schema block pad itself so that the first record batch is
>>>>>> aligned, and is there any problem with doing so?
>>>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>>>> recommended (but not required) by the spec?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Emilio
>>>>>>
>>>>>
Re: buffer alignment (format/java/js)
Posted by Li Jin <ic...@gmail.com>.
Hi Emilio,
> So I think the issue is that we are serializing record batches in a
distributed fashion, and then > concatenating them in the streaming format.
Can you show the code for this?
On Tue, Aug 8, 2017 at 12:35 PM, Emilio Lahr-Vivaz <el...@ccri.com>
wrote:
> So I think the issue is that we are serializing record batches in a
> distributed fashion, and then concatenating them in the streaming format.
> However, the message serialization only aligns the start of the buffers,
> which requires it to know the current absolute offset of the output stream.
> Would there be any problem with padding the end of the message, so any
> single serialized record batch would always be a multiple of 8 bytes?
>
> I've put together a branch that does this, and the existing java tests all
> pass. I'm having some trouble running the integration tests though.
>
> Thanks,
>
> Emilio
>
>
> On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
>
>> Hi Wes,
>>
>> You're right, I just realized that. I think the alignment issue might be
>> in some unrelated code, actually. From what I can tell the the arrow
>> writers are aligning buffers correctly; if not I'll open a bug.
>>
>> Thanks,
>>
>> Emilio
>>
>> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>>
>>> hi Emilio,
>>>
>>> From your description, it isn't clear why 8-byte alignment is causing
>>> a problem (as compare with 64-byte alignment). My understanding is
>>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>>
>>> The starting offset for all buffers should be 8-byte aligned, if not
>>> that is a bug. Could you clarify?
>>>
>>> - Wes
>>>
>>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com>
>>> wrote:
>>>
>>>> After looking at it further, I think only the buffers themselves need
>>>> to be
>>>> aligned, not the metadata and/or schema. Would there be any problem with
>>>> changing the alignment to 64 bytes then?
>>>>
>>>> Thanks,
>>>>
>>>> Emilio
>>>>
>>>>
>>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>>
>>>>> I'm looking into buffer alignment in the java writer classes. Currently
>>>>> some files written with the java streaming writer can't be read due to
>>>>> the
>>>>> javascript TypedArray's restriction that the start offset of the array
>>>>> must
>>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>>> must
>>>>> start on a multiple of 4, Float64Vectors must start on a multiple of 8,
>>>>> etc). From a cursory look at the java writer, I believe that the
>>>>> schema that
>>>>> is written first is not aligned at all, and then each record batch
>>>>> pads out
>>>>> its size to a multiple of 8. So:
>>>>>
>>>>> 1. should the schema block pad itself so that the first record batch is
>>>>> aligned, and is there any problem with doing so?
>>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>>> recommended (but not required) by the spec?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Emilio
>>>>>
>>>>
>>>>
>>
>
Re: buffer alignment (format/java/js)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
So I think the issue is that we are serializing record batches in a
distributed fashion, and then concatenating them in the streaming
format. However, the message serialization only aligns the start of the
buffers, which requires it to know the current absolute offset of the
output stream. Would there be any problem with padding the end of the
message, so any single serialized record batch would always be a
multiple of 8 bytes?
I've put together a branch that does this, and the existing java tests
all pass. I'm having some trouble running the integration tests though.
Thanks,
Emilio
On 08/08/2017 09:18 AM, Emilio Lahr-Vivaz wrote:
> Hi Wes,
>
> You're right, I just realized that. I think the alignment issue might
> be in some unrelated code, actually. From what I can tell the the
> arrow writers are aligning buffers correctly; if not I'll open a bug.
>
> Thanks,
>
> Emilio
>
> On 08/08/2017 09:15 AM, Wes McKinney wrote:
>> hi Emilio,
>>
>> From your description, it isn't clear why 8-byte alignment is causing
>> a problem (as compare with 64-byte alignment). My understanding is
>> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>>
>> The starting offset for all buffers should be 8-byte aligned, if not
>> that is a bug. Could you clarify?
>>
>> - Wes
>>
>> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz
>> <el...@ccri.com> wrote:
>>> After looking at it further, I think only the buffers themselves
>>> need to be
>>> aligned, not the metadata and/or schema. Would there be any problem
>>> with
>>> changing the alignment to 64 bytes then?
>>>
>>> Thanks,
>>>
>>> Emilio
>>>
>>>
>>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>>> I'm looking into buffer alignment in the java writer classes.
>>>> Currently
>>>> some files written with the java streaming writer can't be read due
>>>> to the
>>>> javascript TypedArray's restriction that the start offset of the
>>>> array must
>>>> be a multiple of the data size of the array type (i.e. Int32Vectors
>>>> must
>>>> start on a multiple of 4, Float64Vectors must start on a multiple
>>>> of 8,
>>>> etc). From a cursory look at the java writer, I believe that the
>>>> schema that
>>>> is written first is not aligned at all, and then each record batch
>>>> pads out
>>>> its size to a multiple of 8. So:
>>>>
>>>> 1. should the schema block pad itself so that the first record
>>>> batch is
>>>> aligned, and is there any problem with doing so?
>>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>>> recommended (but not required) by the spec?
>>>>
>>>> Thanks,
>>>>
>>>> Emilio
>>>
>
Re: buffer alignment (format/java/js)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
Hi Wes,
You're right, I just realized that. I think the alignment issue might be
in some unrelated code, actually. From what I can tell the the arrow
writers are aligning buffers correctly; if not I'll open a bug.
Thanks,
Emilio
On 08/08/2017 09:15 AM, Wes McKinney wrote:
> hi Emilio,
>
> From your description, it isn't clear why 8-byte alignment is causing
> a problem (as compare with 64-byte alignment). My understanding is
> that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
>
> The starting offset for all buffers should be 8-byte aligned, if not
> that is a bug. Could you clarify?
>
> - Wes
>
> On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
>> After looking at it further, I think only the buffers themselves need to be
>> aligned, not the metadata and/or schema. Would there be any problem with
>> changing the alignment to 64 bytes then?
>>
>> Thanks,
>>
>> Emilio
>>
>>
>> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>> I'm looking into buffer alignment in the java writer classes. Currently
>>> some files written with the java streaming writer can't be read due to the
>>> javascript TypedArray's restriction that the start offset of the array must
>>> be a multiple of the data size of the array type (i.e. Int32Vectors must
>>> start on a multiple of 4, Float64Vectors must start on a multiple of 8,
>>> etc). From a cursory look at the java writer, I believe that the schema that
>>> is written first is not aligned at all, and then each record batch pads out
>>> its size to a multiple of 8. So:
>>>
>>> 1. should the schema block pad itself so that the first record batch is
>>> aligned, and is there any problem with doing so?
>>> 2. is there any problem with changing the alignment to 64 bytes, as
>>> recommended (but not required) by the spec?
>>>
>>> Thanks,
>>>
>>> Emilio
>>
Re: buffer alignment (format/java/js)
Posted by Wes McKinney <we...@gmail.com>.
hi Emilio,
From your description, it isn't clear why 8-byte alignment is causing
a problem (as compare with 64-byte alignment). My understanding is
that JavaScript's TypedArray classes range in size from 1 to 8 bytes.
The starting offset for all buffers should be 8-byte aligned, if not
that is a bug. Could you clarify?
- Wes
On Tue, Aug 8, 2017 at 8:52 AM, Emilio Lahr-Vivaz <el...@ccri.com> wrote:
> After looking at it further, I think only the buffers themselves need to be
> aligned, not the metadata and/or schema. Would there be any problem with
> changing the alignment to 64 bytes then?
>
> Thanks,
>
> Emilio
>
>
> On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
>>
>> I'm looking into buffer alignment in the java writer classes. Currently
>> some files written with the java streaming writer can't be read due to the
>> javascript TypedArray's restriction that the start offset of the array must
>> be a multiple of the data size of the array type (i.e. Int32Vectors must
>> start on a multiple of 4, Float64Vectors must start on a multiple of 8,
>> etc). From a cursory look at the java writer, I believe that the schema that
>> is written first is not aligned at all, and then each record batch pads out
>> its size to a multiple of 8. So:
>>
>> 1. should the schema block pad itself so that the first record batch is
>> aligned, and is there any problem with doing so?
>> 2. is there any problem with changing the alignment to 64 bytes, as
>> recommended (but not required) by the spec?
>>
>> Thanks,
>>
>> Emilio
>
>
Re: buffer alignment (format/java/js)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
After looking at it further, I think only the buffers themselves need to
be aligned, not the metadata and/or schema. Would there be any problem
with changing the alignment to 64 bytes then?
Thanks,
Emilio
On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
> I'm looking into buffer alignment in the java writer classes.
> Currently some files written with the java streaming writer can't be
> read due to the javascript TypedArray's restriction that the start
> offset of the array must be a multiple of the data size of the array
> type (i.e. Int32Vectors must start on a multiple of 4, Float64Vectors
> must start on a multiple of 8, etc). From a cursory look at the java
> writer, I believe that the schema that is written first is not aligned
> at all, and then each record batch pads out its size to a multiple of
> 8. So:
>
> 1. should the schema block pad itself so that the first record batch
> is aligned, and is there any problem with doing so?
> 2. is there any problem with changing the alignment to 64 bytes, as
> recommended (but not required) by the spec?
>
> Thanks,
>
> Emilio