[GitHub] [beam] mosche opened a new issue, #22156: [Feature Request]: Support more recent versions of Spark 3

[GitBox <gi...@apache.org>]

mosche opened a new issue, #22156:
URL: https://github.com/apache/beam/issues/22156

   ### What would you like to happen?
   
   The Spark 3 runner is currently compiled using Spark 3.1.2 and fails using laters versions.
   Compilation and tests should be fixed to work with Spark 3.2 and 3.3 as well.
   
   As far as known this currently mostly affects the metrics integration (and the structured streaming runner).
   
   Note: Spark 3.2/3.2 also bump their Avro version to 1.10/1.11. This may cause further issues if Avro is used. See
   #19969 and potentially other [issues](https://github.com/apache/beam/issues?q=is%3Aissue+is%3Aopen+label%3Ajava+avro+) 
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: runner-spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche closed issue #22156: [Feature Request]: Support more recent versions of Spark 3

[GitBox <gi...@apache.org>]

mosche closed issue #22156: [Feature Request]: Support more recent versions of Spark 3
URL: https://github.com/apache/beam/issues/22156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org