You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Sri Ganesh Venkataraman <sr...@gmail.com> on 2023/11/03 04:06:14 UTC

[Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

Does Apache Beam version (2.41.0)  or latest (2.51.0) support Spark 3.5
environment for spark runner ?

Apache Beam - Spark Runner Documentation states -
The Spark runner currently supports Spark’s 3.2.x branch

Thanks
Sri Ganesh V

Re: [Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

Posted by Alexey Romanenko <ar...@gmail.com>.

AFAICT, the latest tested (compatibility tests) version for now is 3.4.1 [1] We may try to add 3.5.x version there.

I believe that ValidateRunners tests are run only against default Spark 3.2.2 version.

—
Alexey

[1] https://github.com/apache/beam/blob/2aaf09c0eb6928390d861ba228447338b8ca92d3/runners/spark/3/build.gradle#L36

> On 3 Nov 2023, at 05:06, Sri Ganesh Venkataraman <sr...@gmail.com> wrote:
> 
> Does Apache Beam version (2.41.0)  or latest (2.51.0) support Spark 3.5 environment for spark runner ?
> 
> Apache Beam - Spark Runner Documentation states -
> The Spark runner currently supports Spark’s 3.2.x branch
> 
> Thanks
> Sri Ganesh V