You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dane Pitkin <da...@voltrondata.com.INVALID> on 2023/07/06 18:25:53 UTC

Apache Arrow integration issue with Spark involving Netty

Hi all,

The next release of Apache Arrow v13.0.0 coming this month[1] has upgraded
Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We are seeing
that Spark using Netty v4.1.93.Final is not compatible with Arrow
v13.0.0, throwing an exception at runtime[4]. There has been some talk in a
Spark PR about upgrading to Netty v4.1.94.Final once the new
arrow-memory-netty is released[5].

Should the Spark POM be updated to shade arrow-memory-netty?

Thanks,
Dane

[1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f
[2] https://github.com/apache/arrow/pull/36211
[3] https://github.com/advisories/GHSA-6mjq-h674-j845
[4] https://github.com/apache/arrow/issues/36332
[5] https://github.com/apache/spark/pull/41681

Re: Apache Arrow integration issue with Spark involving Netty

Posted by Dane Pitkin <da...@voltrondata.com.INVALID>.
Update! Netty has reverted the affecting change in v4.1.96. See netty
commit here[1] and arrow PR to upgrade here[2].

The upcoming release of arrow-memory-netty v13 should work with netty
versions <4.1.94 and >=4.1.96.

[1]
https://github.com/netty/netty/commit/dc16c5818a5cd0711f17e0a966783cdc84c9db01
[2] https://github.com/apache/arrow/pull/36926

On Thu, Jul 13, 2023 at 11:47 AM Dane Pitkin <da...@voltrondata.com> wrote:

> I just want to add that there is a Spark Jira issue[1] for upgrading Netty
> once Arrow v13.0.0 is released this month.
>
> [1] https://issues.apache.org/jira/projects/SPARK/issues/SPARK-44212
>
> On Thu, Jul 6, 2023 at 2:25 PM Dane Pitkin <da...@voltrondata.com> wrote:
>
>> Hi all,
>>
>> The next release of Apache Arrow v13.0.0 coming this month[1] has
>> upgraded Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We
>> are seeing that Spark using Netty v4.1.93.Final is not compatible with
>> Arrow v13.0.0, throwing an exception at runtime[4]. There has been some
>> talk in a Spark PR about upgrading to Netty v4.1.94.Final once the new
>> arrow-memory-netty is released[5].
>>
>> Should the Spark POM be updated to shade arrow-memory-netty?
>>
>> Thanks,
>> Dane
>>
>> [1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f
>> [2] https://github.com/apache/arrow/pull/36211
>> [3] https://github.com/advisories/GHSA-6mjq-h674-j845
>> [4] https://github.com/apache/arrow/issues/36332
>> [5] https://github.com/apache/spark/pull/41681
>>
>>

Re: Apache Arrow integration issue with Spark involving Netty

Posted by Dane Pitkin <da...@voltrondata.com.INVALID>.
I just want to add that there is a Spark Jira issue[1] for upgrading Netty
once Arrow v13.0.0 is released this month.

[1] https://issues.apache.org/jira/projects/SPARK/issues/SPARK-44212

On Thu, Jul 6, 2023 at 2:25 PM Dane Pitkin <da...@voltrondata.com> wrote:

> Hi all,
>
> The next release of Apache Arrow v13.0.0 coming this month[1] has upgraded
> Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We are seeing
> that Spark using Netty v4.1.93.Final is not compatible with Arrow
> v13.0.0, throwing an exception at runtime[4]. There has been some talk in a
> Spark PR about upgrading to Netty v4.1.94.Final once the new
> arrow-memory-netty is released[5].
>
> Should the Spark POM be updated to shade arrow-memory-netty?
>
> Thanks,
> Dane
>
> [1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f
> [2] https://github.com/apache/arrow/pull/36211
> [3] https://github.com/advisories/GHSA-6mjq-h674-j845
> [4] https://github.com/apache/arrow/issues/36332
> [5] https://github.com/apache/spark/pull/41681
>
>