You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/24 12:08:41 UTC
[GitHub] [beam] aromanenko-dev opened a new pull request, #23805: Bump Spark3 version to 3.2.0
aromanenko-dev opened a new pull request, #23805:
URL: https://github.com/apache/beam/pull/23805
Bump Spark3 version to 3.2.0
Closes #23804
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309096386
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1642262979
Run Python_Integration PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288941390
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1634288859
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1582512500
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1318678902
Stopping reviewer notifications for this pull request: requested by reviewer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288941646
Run Spark ValidatesRunner Java 11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1642263782
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309062253
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1318677367
stop reviewer notifications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1642338407
@mosche ping on this
As we discussed privately, hadoop-related things are not so important in this case and could be addressed out of this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev merged PR #23805:
URL: https://github.com/apache/beam/pull/23805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1318548067
Reminder, please take a look at this pr: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288940931
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #23805:
URL: https://github.com/apache/beam/pull/23805#discussion_r1003258810
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -494,7 +494,7 @@ class BeamModulePlugin implements Plugin<Project> {
def sbe_tool_version = "1.25.1"
def slf4j_version = "1.7.30"
def spark2_version = "2.4.8"
- def spark3_version = "3.1.2"
+ def spark3_version = "3.2.0"
Review Comment:
Wow, tests are failing... looks like 3.2.0 to 3.2.1 (which is currently tested as part of `sparkVersionsTest`) contains a breaking change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1295053931
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288957664
Run Spark Runner Tpcds Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1476519817
@aromanenko-dev Shall we pick this up again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1634288071
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309072362
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1643714263
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23805: Bump Spark3 version to 3.2.2
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #23805:
URL: https://github.com/apache/beam/pull/23805#discussion_r1268974122
##########
website/www/site/content/en/documentation/runners/spark.md:
##########
@@ -67,8 +67,8 @@ the portable Runner. For more information on portability, please visit the
## Spark Runner prerequisites and setup
-The Spark runner currently supports Spark's 3.1.x branch.
-> **Note:** Support for Spark 2.4.x was deprecated as of Beam 2.41.0 and finally dropped with the release of Beam 2.46.0.
+The Spark runner currently supports Spark's 3.2.x branch.
+> **Note:** Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 (or soon after).
Review Comment:
```suggestion
> **Note:** Support for Spark 2.4.x was dropped with Beam 2.46.0.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1475243160
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1476560373
@mosche Yes, I was thinking to finish this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309119361
@mosche All tests are passing now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1642263518
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23805: Bump Spark3 version to 3.2.2
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #23805:
URL: https://github.com/apache/beam/pull/23805#discussion_r1268974914
##########
runners/spark/spark_runner.gradle:
##########
@@ -181,7 +181,13 @@ dependencies {
runtimeOnly library.java.jackson_module_scala_2_12
// Force paranamer 2.8 to avoid issues when using Scala 2.12
runtimeOnly "com.thoughtworks.paranamer:paranamer:2.8"
- provided library.java.hadoop_common
+ if (project.property("spark_version").startsWith("3.2")) {
Review Comment:
Spark 2.4 was dropped, it's fine to remove the if clause here and just keep hadoop-client-api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1643785209
Run SQL PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #23805:
URL: https://github.com/apache/beam/pull/23805#discussion_r1003239896
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -494,7 +494,7 @@ class BeamModulePlugin implements Plugin<Project> {
def sbe_tool_version = "1.25.1"
def slf4j_version = "1.7.30"
def spark2_version = "2.4.8"
- def spark3_version = "3.1.2"
+ def spark3_version = "3.2.0"
Review Comment:
Why not 3.2.2?
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -494,7 +494,7 @@ class BeamModulePlugin implements Plugin<Project> {
def sbe_tool_version = "1.25.1"
def slf4j_version = "1.7.30"
def spark2_version = "2.4.8"
- def spark3_version = "3.1.2"
+ def spark3_version = "3.2.0"
Review Comment:
Btw 3.2.0 / 3.2.1 contain CVEs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1385373793
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1643714458
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.2
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1643784713
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288986899
# [Codecov](https://codecov.io/gh/apache/beam/pull/23805?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23805](https://codecov.io/gh/apache/beam/pull/23805?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fc88548) into [master](https://codecov.io/gh/apache/beam/commit/760c83e94020971d84b904270b087d4c3d69fd00?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (760c83e) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #23805 +/- ##
=======================================
Coverage 73.19% 73.19%
=======================================
Files 720 720
Lines 96069 96069
=======================================
Hits 70313 70313
Misses 24444 24444
Partials 1312 1312
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1288963469
I'd also suggest to document this change in CHANGES.md. Particularly for anyone using the job-server, this is a breaking change as they have to upgrade their cluster version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309062737
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23805: Bump Spark3 version to 3.2.0
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23805:
URL: https://github.com/apache/beam/pull/23805#issuecomment-1309794612
@aromanenko-dev a few things are still left
- Please update the Spark version mentioned in the Spark runner docs. Also this change should be mentioned in CHANGES.md https://github.com/apache/beam/blame/master/website/www/site/content/en/documentation/runners/spark.md#L70
- What about the Spark versions used for compatibility tests? It currently contains 3.2.1, if we bump to 3.2.0 we should keep that. But what about 3.1.1? Should we add it there to ensure we stay compatible?
https://github.com/apache/beam/blob/master/runners/spark/3/build.gradle#L36-L39
- The hadoop version tests won't reliably work work anymore as both dependencies are pulled. So it's unclear what gets actually used.
```
hadoopVersions.each { kv ->
configurations."hadoopVersion$kv.key" {
resolutionStrategy {
force "org.apache.hadoop:hadoop-common:$kv.value"
}
}
}
```
- Similarly, VR tests are pulling hadoop-format as dependency and it's not clear what version of hadoop is effectively used to run the tests. We have to make sure we run tests enforcing the version used by Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org