Status of Rust Integration Testing

[Neville Dipale <ne...@gmail.com>]

Good day Arrow devs,

I've spent a few evenings looking into the issues that we're experiencing
with Rust integration testing.
In summary, none of our tests pass (zero batch doesn't count :) ).
This is mainly because of changes from the legacy padding in the 0.15.0
release, which we never made in Rust (
https://issues.apache.org/jira/browse/ARROW-6313).
I honestly didn't see this at the time.

Anyways, the implication is that Rust is using the 'legacy' format of
4-byte alignment, and fails to parse and read the ipc::Message into
RecordBatches.

I'm so far struggling with the fixes in Rust, so this work might take long
and likely won't make it to 1.0.0. I'll open JIRAs for whatever work is
necessary.

The Rust implementation appears to be behind with quite some implementation
details, so I'll try catching up by the next release.

Regards
Neville

Re: Status of Rust Integration Testing

[Neville Dipale <ne...@gmail.com>]

Hi Micah,

Yes, those files are read correctly. We test against them.
I was trying to generate gold files based on 0.17.1, so I could debug
against those, I'll work on that in the coming days.

On Sat, 11 Jul 2020, 05:58 Micah Kornfield, <em...@gmail.com> wrote:

> Hi Neville,
> Thanks for the update.  One question, we have "gold" files for 0.14.0
> checked into the test-data repo and run integration tests on those to
> ensure we can read them in a few implementations.  Does Rust at least read
> those correctly?
>
> Thanks,
> Micah
>
> On Fri, Jul 10, 2020 at 1:03 PM Neville Dipale <ne...@gmail.com>
> wrote:
>
> > Good day Arrow devs,
> >
> > I've spent a few evenings looking into the issues that we're experiencing
> > with Rust integration testing.
> > In summary, none of our tests pass (zero batch doesn't count :) ).
> > This is mainly because of changes from the legacy padding in the 0.15.0
> > release, which we never made in Rust (
> > https://issues.apache.org/jira/browse/ARROW-6313).
> > I honestly didn't see this at the time.
> >
> > Anyways, the implication is that Rust is using the 'legacy' format of
> > 4-byte alignment, and fails to parse and read the ipc::Message into
> > RecordBatches.
> >
> > I'm so far struggling with the fixes in Rust, so this work might take
> long
> > and likely won't make it to 1.0.0. I'll open JIRAs for whatever work is
> > necessary.
> >
> > The Rust implementation appears to be behind with quite some
> implementation
> > details, so I'll try catching up by the next release.
> >
> > Regards
> > Neville
> >
>

Re: Status of Rust Integration Testing

[Micah Kornfield <em...@gmail.com>]

Hi Neville,
Thanks for the update.  One question, we have "gold" files for 0.14.0
checked into the test-data repo and run integration tests on those to
ensure we can read them in a few implementations.  Does Rust at least read
those correctly?

Thanks,
Micah

On Fri, Jul 10, 2020 at 1:03 PM Neville Dipale <ne...@gmail.com>
wrote:

> Good day Arrow devs,
>
> I've spent a few evenings looking into the issues that we're experiencing
> with Rust integration testing.
> In summary, none of our tests pass (zero batch doesn't count :) ).
> This is mainly because of changes from the legacy padding in the 0.15.0
> release, which we never made in Rust (
> https://issues.apache.org/jira/browse/ARROW-6313).
> I honestly didn't see this at the time.
>
> Anyways, the implication is that Rust is using the 'legacy' format of
> 4-byte alignment, and fails to parse and read the ipc::Message into
> RecordBatches.
>
> I'm so far struggling with the fixes in Rust, so this work might take long
> and likely won't make it to 1.0.0. I'll open JIRAs for whatever work is
> necessary.
>
> The Rust implementation appears to be behind with quite some implementation
> details, so I'll try catching up by the next release.
>
> Regards
> Neville
>