[Spark] spark client for Hadoop 2.x

[Amin Borjian <bo...@outlook.com>]

From Spark version 3.1.0 onwards, the clients provided for Spark are built with Hadoop 3 and placed in maven Repository. Unfortunately  we use Hadoop 2.7.7 in our infrastructure currently.

1) Does Spark have a plan to publish the Spark client dependencies for Hadoop 2.x?
2) Are the new Spark clients capable of connecting to the Hadoop 2.x cluster? (According to a simple test, Spark client 3.2.1 had no problem with the Hadoop 2.7 cluster but we wanted to know if there was any guarantee from Spark?)

Thank you very much in advance
Amin Borjian

Re: [Spark] spark client for Hadoop 2.x

[Morven Huang <mo...@gmail.com>]

I remember that ./dev/make-distribution.sh in spark source allows people to specify Hadoop version.

> 2022年4月6日 下午4:31，Amin Borjian <bo...@outlook.com> 写道：
> 
> From Spark version 3.1.0 onwards, the clients provided for Spark are built with Hadoop 3 and placed in maven Repository. Unfortunately  we use Hadoop 2.7.7 in our infrastructure currently.
>  
> 1) Does Spark have a plan to publish the Spark client dependencies for Hadoop 2.x?
> 2) Are the new Spark clients capable of connecting to the Hadoop 2.x cluster? (According to a simple test, Spark client 3.2.1 had no problem with the Hadoop 2.7 cluster but we wanted to know if there was any guarantee from Spark?)
>  
> Thank you very much in advance
> Amin Borjian