You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Adam Hooper <ad...@adamhooper.com> on 2019/12/18 16:42:38 UTC
Where can I report a security-related issue?
My project parses Arrow files produced by untrusted code.
It looks to me like the "validate" function should help me avoid undefined
behavior given an invalid Arrow file. I found a bug in the function: even
after validation, an invalid Arrow file can trigger undefined behavior.
Is security a goal of the Arrow project/format? If so, how shall I report
this bug without endangering other users in my situation?
Enjoy life,
Adam
--
Adam Hooper
+1-514-882-9694
http://adamhooper.com
Re: Where can I report a security-related issue?
Posted by Wes McKinney <we...@gmail.com>.
hi Adam,
Can you please send the security issue to private@arrow.apache.org
(this is a moderated non-public list) and we can discuss there.
Thanks,
Wes
On Wed, Dec 18, 2019 at 10:43 AM Adam Hooper <ad...@adamhooper.com> wrote:
>
> My project parses Arrow files produced by untrusted code.
>
> It looks to me like the "validate" function should help me avoid undefined
> behavior given an invalid Arrow file. I found a bug in the function: even
> after validation, an invalid Arrow file can trigger undefined behavior.
>
> Is security a goal of the Arrow project/format? If so, how shall I report
> this bug without endangering other users in my situation?
>
> Enjoy life,
> Adam
>
> --
> Adam Hooper
> +1-514-882-9694
> http://adamhooper.com
Re: Where can I report a security-related issue?
Posted by Adam Hooper <ad...@adamhooper.com>.
On Thu, Dec 19, 2019 at 8:43 AM Antoine Pitrou <an...@python.org> wrote:
>
> Security is a concern for us and we welcome any reports. That said,
> Arrow is primarily concerned about efficiency (memory and processing
> efficiency), which implies that implicit checks cannot often be
> comprehensive.
>
Thank you for clarifying. This is all music to my ears. I feel Arrow's
careful design gives me all the tools I need to confidently repel malicious
input.
<https://github.com/cyb70289/utf8>
Enjoy life,
Adam
--
Adam Hooper
+1-514-882-9694
http://adamhooper.com
Re: Where can I report a security-related issue?
Posted by Antoine Pitrou <an...@python.org>.
Hi Adam,
Le 18/12/2019 à 20:00, Adam Hooper a écrit :
> Thank you for the responses, Wes and Antoine. I've sent to private@ and
> then filed https://issues.apache.org/jira/browse/ARROW-7435
>
> @Antoine I'm glad to hear the IPC layer is a security boundary. Could you
> please clarify what that means exactly? Do you mean (please check all that
> apply):
What I meant is that ill-formed IPC streams should never cause Arrow to
crash or behave erratically when reading, but instead return an error.
However, a well-formed IPC stream can still contain ill-formed data, as
you experienced.
> 1. Arrow IPC is designed to produce defined behavior given any input?
> 2. Arrow IPC is designed such that a nefarious Arrow file can't cause reads
> or writes outside of the file's memory boundaries?
> 3. If I notice a specially-crafted file inspires Arrow IPC to violate these
> first two design goals, that's a bug I should report ASAP?
Ideally, yes to all three, but with the limitation that those principles
only apply to the IPC operation per se (the operation of writing or
reading the data from/to memory). The IPC layer won't detect that a
serialized list array has indices out of bounds in its offsets buffer,
for example.
We also have a basic fuzzing setup for the C++ IPC implementation,
though it currently doesn't get the attention it deserves. We will try
to improve that in the future (see
https://issues.apache.org/jira/browse/ARROW-7433 : "Set up OSS-Fuzz").
> In brief: I'm asking whether security is a project goal or more of a
> pleasant-but-unsupported consequence of Arrow's design. *Should* I use
> Arrow IPC (with Validate()) to read files produced by untrusted
> (out-of-process) code?
Security is a concern for us and we welcome any reports. That said,
Arrow is primarily concerned about efficiency (memory and processing
efficiency), which implies that implicit checks cannot often be
comprehensive.
In git master, Arrow C++ has two validation routines. None of them are
called by default, you have to call them yourself:
- Validate() does cheap checks (O(1) in array length), which are mostly
metadata checks
- ValidateFull() does more comprehensive data checks (potentially O(N)
in array length)
If some ill-formed data doesn't get caught by ValidateFull(), you should
definitely open a bug report on JIRA. We also welcome suggestions for
additional checks in Validate(), provided they are not dependent on
array length.
Regards
Antoine.
Re: Where can I report a security-related issue?
Posted by Adam Hooper <ad...@adamhooper.com>.
Thank you for the responses, Wes and Antoine. I've sent to private@ and
then filed https://issues.apache.org/jira/browse/ARROW-7435
@Antoine I'm glad to hear the IPC layer is a security boundary. Could you
please clarify what that means exactly? Do you mean (please check all that
apply):
1. Arrow IPC is designed to produce defined behavior given any input?
2. Arrow IPC is designed such that a nefarious Arrow file can't cause reads
or writes outside of the file's memory boundaries?
3. If I notice a specially-crafted file inspires Arrow IPC to violate these
first two design goals, that's a bug I should report ASAP?
In brief: I'm asking whether security is a project goal or more of a
pleasant-but-unsupported consequence of Arrow's design. *Should* I use
Arrow IPC (with Validate()) to read files produced by untrusted
(out-of-process) code?
Enjoy life,
Adam
On Wed, Dec 18, 2019 at 12:30 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hi Adam,
>
> The validation was significantly enhanced in git master. In particular,
> Array::ValidateFull() will validate data more or less extensively. I
> wouldn't be surprised if not everything is validated, though. Feel free
> to open JIRA tickets about missing validation checks.
>
> However, if you are in control of the current process, then you can
> pretty much craft an invalid array in all kinds of manners. The main
> security boundary is the IPC layer and other file format implementations
> (CSV, Parquet...).
>
> Regards
>
> Antoine.
>
>
> Le 18/12/2019 à 17:42, Adam Hooper a écrit :
> > My project parses Arrow files produced by untrusted code.
> >
> > It looks to me like the "validate" function should help me avoid
> undefined
> > behavior given an invalid Arrow file. I found a bug in the function: even
> > after validation, an invalid Arrow file can trigger undefined behavior.
> >
> > Is security a goal of the Arrow project/format? If so, how shall I report
> > this bug without endangering other users in my situation?
> >
> > Enjoy life,
> > Adam
> >
>
--
Adam Hooper
+1-514-882-9694
http://adamhooper.com
Re: Where can I report a security-related issue?
Posted by Antoine Pitrou <an...@python.org>.
Hi Adam,
The validation was significantly enhanced in git master. In particular,
Array::ValidateFull() will validate data more or less extensively. I
wouldn't be surprised if not everything is validated, though. Feel free
to open JIRA tickets about missing validation checks.
However, if you are in control of the current process, then you can
pretty much craft an invalid array in all kinds of manners. The main
security boundary is the IPC layer and other file format implementations
(CSV, Parquet...).
Regards
Antoine.
Le 18/12/2019 à 17:42, Adam Hooper a écrit :
> My project parses Arrow files produced by untrusted code.
>
> It looks to me like the "validate" function should help me avoid undefined
> behavior given an invalid Arrow file. I found a bug in the function: even
> after validation, an invalid Arrow file can trigger undefined behavior.
>
> Is security a goal of the Arrow project/format? If so, how shall I report
> this bug without endangering other users in my situation?
>
> Enjoy life,
> Adam
>