You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2021/10/01 07:18:30 UTC
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
In C++ we have dedicated validation routines that can be used against
untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
validate that invalid input cannot crash the IPC reader.
(*)
https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
> I have created a WIP PR for initial feedback on the approach of validating
> ArrayData upon creation[1]. If there are no objections to the approach I
> will complete the implementation over the next few days
>
> The approach that Sergey describes of `get` and `unsafe get_unchecked`
> sounds like a good one to me if performance testing shows we need a bypass.
>
> Andrew
>
> [1] https://github.com/apache/arrow-rs/pull/810
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
Posted by Andrew Lamb <al...@influxdata.com>.
To finish up this thread, the rustsec advisory has been updated:
https://github.com/rustsec/advisory-db/pull/1131
On Tue, Nov 30, 2021 at 5:10 PM Andrew Lamb <al...@influxdata.com> wrote:
> Well, I finally have a PR[1] that adds full ArrayData validation ready
> for review Any thoughts or review comments would be most helpful.
>
> Thank you,
> Andrew
>
> [1] https://github.com/apache/arrow-rs/pull/921
>
> On Fri, Oct 29, 2021 at 6:23 AM Andrew Lamb <al...@influxdata.com> wrote:
>
>> There is more discussion about the RUSTSEC process here:
>> https://github.com/rustsec/advisory-db/issues/1092
>>
>> On Wed, Oct 6, 2021 at 10:52 AM Andrew Lamb <al...@influxdata.com> wrote:
>>
>>> I have incorporated feedback into a proposal [1] of how to handle
>>> validation of arguments to ArrayData::new, and would appreciate further
>>> review
>>>
>>> [1] https://github.com/apache/arrow-rs/issues/817
>>>
>>> On Fri, Oct 1, 2021 at 6:44 AM Andrew Lamb <al...@influxdata.com> wrote:
>>>
>>>> Thank you Antoine,
>>>>
>>>> The C++ validation routine is super helpful, especially with respect to
>>>> validating nested structures. I will follow its lead
>>>>
>>>> Andrew
>>>>
>>>> On Fri, Oct 1, 2021 at 3:25 AM Antoine Pitrou <an...@python.org>
>>>> wrote:
>>>>
>>>>>
>>>>> In C++ we have dedicated validation routines that can be used against
>>>>> untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
>>>>> validate that invalid input cannot crash the IPC reader.
>>>>>
>>>>> (*)
>>>>>
>>>>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
>>>>>
>>>>>
>>>>> Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
>>>>> > I have created a WIP PR for initial feedback on the approach of
>>>>> validating
>>>>> > ArrayData upon creation[1]. If there are no objections to the
>>>>> approach I
>>>>> > will complete the implementation over the next few days
>>>>> >
>>>>> > The approach that Sergey describes of `get` and `unsafe
>>>>> get_unchecked`
>>>>> > sounds like a good one to me if performance testing shows we need a
>>>>> bypass.
>>>>> >
>>>>> > Andrew
>>>>> >
>>>>> > [1] https://github.com/apache/arrow-rs/pull/810
>>>>>
>>>>
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
Posted by Andrew Lamb <al...@influxdata.com>.
Well, I finally have a PR[1] that adds full ArrayData validation ready for
review Any thoughts or review comments would be most helpful.
Thank you,
Andrew
[1] https://github.com/apache/arrow-rs/pull/921
On Fri, Oct 29, 2021 at 6:23 AM Andrew Lamb <al...@influxdata.com> wrote:
> There is more discussion about the RUSTSEC process here:
> https://github.com/rustsec/advisory-db/issues/1092
>
> On Wed, Oct 6, 2021 at 10:52 AM Andrew Lamb <al...@influxdata.com> wrote:
>
>> I have incorporated feedback into a proposal [1] of how to handle
>> validation of arguments to ArrayData::new, and would appreciate further
>> review
>>
>> [1] https://github.com/apache/arrow-rs/issues/817
>>
>> On Fri, Oct 1, 2021 at 6:44 AM Andrew Lamb <al...@influxdata.com> wrote:
>>
>>> Thank you Antoine,
>>>
>>> The C++ validation routine is super helpful, especially with respect to
>>> validating nested structures. I will follow its lead
>>>
>>> Andrew
>>>
>>> On Fri, Oct 1, 2021 at 3:25 AM Antoine Pitrou <an...@python.org>
>>> wrote:
>>>
>>>>
>>>> In C++ we have dedicated validation routines that can be used against
>>>> untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
>>>> validate that invalid input cannot crash the IPC reader.
>>>>
>>>> (*)
>>>>
>>>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
>>>>
>>>>
>>>> Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
>>>> > I have created a WIP PR for initial feedback on the approach of
>>>> validating
>>>> > ArrayData upon creation[1]. If there are no objections to the
>>>> approach I
>>>> > will complete the implementation over the next few days
>>>> >
>>>> > The approach that Sergey describes of `get` and `unsafe get_unchecked`
>>>> > sounds like a good one to me if performance testing shows we need a
>>>> bypass.
>>>> >
>>>> > Andrew
>>>> >
>>>> > [1] https://github.com/apache/arrow-rs/pull/810
>>>>
>>>
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
Posted by Andrew Lamb <al...@influxdata.com>.
There is more discussion about the RUSTSEC process here:
https://github.com/rustsec/advisory-db/issues/1092
On Wed, Oct 6, 2021 at 10:52 AM Andrew Lamb <al...@influxdata.com> wrote:
> I have incorporated feedback into a proposal [1] of how to handle
> validation of arguments to ArrayData::new, and would appreciate further
> review
>
> [1] https://github.com/apache/arrow-rs/issues/817
>
> On Fri, Oct 1, 2021 at 6:44 AM Andrew Lamb <al...@influxdata.com> wrote:
>
>> Thank you Antoine,
>>
>> The C++ validation routine is super helpful, especially with respect to
>> validating nested structures. I will follow its lead
>>
>> Andrew
>>
>> On Fri, Oct 1, 2021 at 3:25 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>>
>>> In C++ we have dedicated validation routines that can be used against
>>> untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
>>> validate that invalid input cannot crash the IPC reader.
>>>
>>> (*)
>>>
>>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
>>>
>>>
>>> Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
>>> > I have created a WIP PR for initial feedback on the approach of
>>> validating
>>> > ArrayData upon creation[1]. If there are no objections to the approach
>>> I
>>> > will complete the implementation over the next few days
>>> >
>>> > The approach that Sergey describes of `get` and `unsafe get_unchecked`
>>> > sounds like a good one to me if performance testing shows we need a
>>> bypass.
>>> >
>>> > Andrew
>>> >
>>> > [1] https://github.com/apache/arrow-rs/pull/810
>>>
>>
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
Posted by Andrew Lamb <al...@influxdata.com>.
I have incorporated feedback into a proposal [1] of how to handle
validation of arguments to ArrayData::new, and would appreciate further
review
[1] https://github.com/apache/arrow-rs/issues/817
On Fri, Oct 1, 2021 at 6:44 AM Andrew Lamb <al...@influxdata.com> wrote:
> Thank you Antoine,
>
> The C++ validation routine is super helpful, especially with respect to
> validating nested structures. I will follow its lead
>
> Andrew
>
> On Fri, Oct 1, 2021 at 3:25 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> In C++ we have dedicated validation routines that can be used against
>> untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
>> validate that invalid input cannot crash the IPC reader.
>>
>> (*)
>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
>>
>>
>> Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
>> > I have created a WIP PR for initial feedback on the approach of
>> validating
>> > ArrayData upon creation[1]. If there are no objections to the approach I
>> > will complete the implementation over the next few days
>> >
>> > The approach that Sergey describes of `get` and `unsafe get_unchecked`
>> > sounds like a good one to me if performance testing shows we need a
>> bypass.
>> >
>> > Andrew
>> >
>> > [1] https://github.com/apache/arrow-rs/pull/810
>>
>
Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs
Posted by Andrew Lamb <al...@influxdata.com>.
Thank you Antoine,
The C++ validation routine is super helpful, especially with respect to
validating nested structures. I will follow its lead
Andrew
On Fri, Oct 1, 2021 at 3:25 AM Antoine Pitrou <an...@python.org> wrote:
>
> In C++ we have dedicated validation routines that can be used against
> untrusted input. (*) We also have fuzzing set up using OSS-Fuzz to
> validate that invalid input cannot crash the IPC reader.
>
> (*)
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/validate.h
>
>
> Le 01/10/2021 à 00:13, Andrew Lamb a écrit :
> > I have created a WIP PR for initial feedback on the approach of
> validating
> > ArrayData upon creation[1]. If there are no objections to the approach I
> > will complete the implementation over the next few days
> >
> > The approach that Sergey describes of `get` and `unsafe get_unchecked`
> > sounds like a good one to me if performance testing shows we need a
> bypass.
> >
> > Andrew
> >
> > [1] https://github.com/apache/arrow-rs/pull/810
>