[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.0

[GitBox <gi...@apache.org>]

mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309794612

   @aromanenko-dev a few things are still left
   
   - Please update the Spark version mentioned in the Spark runner docs. Also this change should be mentioned in CHANGES.md https://github.com/apache/beam/blame/master/website/www/site/content/en/documentation/runners/spark.md#L70
   
   - What about the Spark versions used for compatibility tests? It currently contains 3.2.1, if we bump to 3.2.0 we should keep that. But what about 3.1.1? Should we add it there to ensure we stay compatible? 
     https://github.com/apache/beam/blob/master/runners/spark/3/build.gradle#L36-L39
   
   - The hadoop version tests won't reliably work work anymore as both dependencies are pulled. So it's unclear what gets actually used.
       ```
       hadoopVersions.each { kv ->
         configurations."hadoopVersion$kv.key" {
           resolutionStrategy {
             force "org.apache.hadoop:hadoop-common:$kv.value"
           }
         }
       }
       ```
   
   -  Similarly, VR tests are pulling hadoop-format as dependency and it's not clear what version of hadoop is effectively used to run the tests. We have to make sure we run tests enforcing the version used by Spark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org