You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by GitBox <gi...@apache.org> on 2020/03/31 16:26:05 UTC
[GitHub] [arrow-site] pitrou opened a new pull request #49: ARROW-7847:
[Website] Add blog post about fuzzing the IPC layer
pitrou opened a new pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] fsaintjacques commented on a change in pull request
#49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#discussion_r402499744
##########
File path: _posts/2020-04-01-fuzzing-arrow-ipc.md
##########
@@ -0,0 +1,89 @@
+---
+layout: post
+title: "Fuzzing the Arrow C++ IPC implementation"
+description: "We have set up continuous fuzzing for the Arrow C++ IPC reader.
+This helped us find and correct several issues where missing input validation
+would lead to crashes or undefined behaviour."
+date: "2020-04-01 00:00:00 +0100"
+author: apitrou
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Apache Arrow aims to allow fast and seamless data interchange between
+heterogenous runtimes and environments. Whether using the columnar
+[IPC stream protocol](https://arrow.apache.org/docs/format/Columnar.html),
+the [Flight](https://arrow.apache.org/docs/format/Flight.html) RPC layer,
+the Feather file format, the
+[Plasma](https://arrow.apache.org/docs/python/plasma.html) shared object
+store, or any application-specific data distribution mechanism, Arrow IPC
+implementations may try to decode data from untrusted input. While it is ok
+to report an error in that case, Arrow shouldn't crash or engage in risky
+behaviour while reading such data.
+
+To validate the robustness of the Arrow C++ IPC reader (which also underlies
+the Python, C/GLib, R and Ruby bindings), we
+[successfully submitted](https://github.com/google/oss-fuzz/pull/3233)
+the Arrow project to OSS-Fuzz, a continuous fuzzing initiative for critical
+open source projects, provided by Google.
+
+## What is being fuzzed
+
+As of this writing, the `RecordBatchStreamReader` and `RecordBatchFileReader`
Review comment:
I'd add that ParquetReader are also fuzzed since this is one of the main reason to use arrow (even if the title is about IPC).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou merged pull request #49: ARROW-7847: [Website]
Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou merged pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] wesm commented on a change in pull request #49:
ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#discussion_r401264928
##########
File path: _posts/2020-04-xx-fuzzing-arrow-ipc.md
##########
@@ -0,0 +1,88 @@
+---
+layout: post
+title: "Fuzzing the Arrow C++ IPC implementation"
+description: "We have set up continuous fuzzing for the Arrow C++ IPC reader.
+This helped us find and correct several issues where missing input validation
+would lead to crashes or undefined behaviour."
+date: "2020-04-01 00:00:00 +0100"
+author: apitrou
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Apache Arrow aims to allow fast and seamless data interchange between
+heterogenous runtimes and environments. Whether using the
+[Flight](https://arrow.apache.org/docs/format/Flight.html) RPC layer,
+the Feather file format, the
Review comment:
Perhaps add the "the columnar IPC stream protocol" to this list, since this is used in a bunch of Arrow-based interchange setups (e.g. Spark, various database protocols, etc.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou commented on a change in pull request #49:
ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#discussion_r401549303
##########
File path: _posts/2020-04-xx-fuzzing-arrow-ipc.md
##########
@@ -0,0 +1,88 @@
+---
+layout: post
+title: "Fuzzing the Arrow C++ IPC implementation"
+description: "We have set up continuous fuzzing for the Arrow C++ IPC reader.
+This helped us find and correct several issues where missing input validation
+would lead to crashes or undefined behaviour."
+date: "2020-04-01 00:00:00 +0100"
+author: apitrou
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Apache Arrow aims to allow fast and seamless data interchange between
+heterogenous runtimes and environments. Whether using the
+[Flight](https://arrow.apache.org/docs/format/Flight.html) RPC layer,
+the Feather file format, the
Review comment:
Done, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou commented on issue #49: ARROW-7847: [Website]
Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#issuecomment-610529000
Oops, sorry. I'll try to next time :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] wesm commented on a change in pull request #49:
ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#discussion_r402601481
##########
File path: _posts/2020-04-01-fuzzing-arrow-ipc.md
##########
@@ -0,0 +1,89 @@
+---
+layout: post
+title: "Fuzzing the Arrow C++ IPC implementation"
+description: "We have set up continuous fuzzing for the Arrow C++ IPC reader.
+This helped us find and correct several issues where missing input validation
+would lead to crashes or undefined behaviour."
+date: "2020-04-01 00:00:00 +0100"
+author: apitrou
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Apache Arrow aims to allow fast and seamless data interchange between
+heterogenous runtimes and environments. Whether using the columnar
+[IPC stream protocol](https://arrow.apache.org/docs/format/Columnar.html),
+the [Flight](https://arrow.apache.org/docs/format/Flight.html) RPC layer,
+the Feather file format, the
+[Plasma](https://arrow.apache.org/docs/python/plasma.html) shared object
+store, or any application-specific data distribution mechanism, Arrow IPC
+implementations may try to decode data from untrusted input. While it is ok
+to report an error in that case, Arrow shouldn't crash or engage in risky
+behaviour while reading such data.
+
+To validate the robustness of the Arrow C++ IPC reader (which also underlies
+the Python, C/GLib, R and Ruby bindings), we
+[successfully submitted](https://github.com/google/oss-fuzz/pull/3233)
+the Arrow project to OSS-Fuzz, a continuous fuzzing initiative for critical
+open source projects, provided by Google.
+
+## What is being fuzzed
+
+As of this writing, the `RecordBatchStreamReader` and `RecordBatchFileReader`
Review comment:
It's mentioned at the end of the doc, but with the caveat that there are still various unfixed issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou commented on issue #49: ARROW-7847: [Website]
Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#issuecomment-607368214
@nealrichardson Do you want to add something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou commented on issue #49: ARROW-7847: [Website]
Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#issuecomment-610529189
Ah, I hadn't realized there's a merge script in this repo as well. Sorry!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] pitrou commented on a change in pull request #49:
ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#discussion_r402873609
##########
File path: _posts/2020-04-01-fuzzing-arrow-ipc.md
##########
@@ -0,0 +1,89 @@
+---
+layout: post
+title: "Fuzzing the Arrow C++ IPC implementation"
+description: "We have set up continuous fuzzing for the Arrow C++ IPC reader.
+This helped us find and correct several issues where missing input validation
+would lead to crashes or undefined behaviour."
+date: "2020-04-01 00:00:00 +0100"
+author: apitrou
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Apache Arrow aims to allow fast and seamless data interchange between
+heterogenous runtimes and environments. Whether using the columnar
+[IPC stream protocol](https://arrow.apache.org/docs/format/Columnar.html),
+the [Flight](https://arrow.apache.org/docs/format/Flight.html) RPC layer,
+the Feather file format, the
+[Plasma](https://arrow.apache.org/docs/python/plasma.html) shared object
+store, or any application-specific data distribution mechanism, Arrow IPC
+implementations may try to decode data from untrusted input. While it is ok
+to report an error in that case, Arrow shouldn't crash or engage in risky
+behaviour while reading such data.
+
+To validate the robustness of the Arrow C++ IPC reader (which also underlies
+the Python, C/GLib, R and Ruby bindings), we
+[successfully submitted](https://github.com/google/oss-fuzz/pull/3233)
+the Arrow project to OSS-Fuzz, a continuous fuzzing initiative for critical
+open source projects, provided by Google.
+
+## What is being fuzzed
+
+As of this writing, the `RecordBatchStreamReader` and `RecordBatchFileReader`
Review comment:
Yes, I hesitate putting it forward as we haven't fixed all issues yet (also, as you mentioned, not all encodings may be covered currently: someone should check). We can publish another post when we feel it is finally stable on that front.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [arrow-site] wesm commented on issue #49: ARROW-7847: [Website] Add
blog post about fuzzing the IPC layer
Posted by GitBox <gi...@apache.org>.
wesm commented on issue #49: ARROW-7847: [Website] Add blog post about fuzzing the IPC layer
URL: https://github.com/apache/arrow-site/pull/49#issuecomment-610528391
If you use the GitHub UI to merge can you squash?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services