You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/12 14:52:42 UTC
[GitHub] [beam] mosche opened a new pull request, #23603: Keep Spark version in a single place only (BeamModulePlugin)
mosche opened a new pull request, #23603:
URL: https://github.com/apache/beam/pull/23603
It's a bit of a trap to have the Spark version in two places.
Especially because it's most obvious to change the one in the Spark build.gradle. But that one is always overruled by the one used in the artifact library in BeamModulePlugin.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a diff in pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on code in PR #23603:
URL: https://github.com/apache/beam/pull/23603#discussion_r998316151
##########
runners/spark/2/build.gradle:
##########
@@ -19,8 +19,8 @@
def basePath = '..'
/* All properties required for loading the Spark build script */
project.ext {
- // Set the version of all Spark-related dependencies here.
- spark_version = '2.4.8'
+ // Spark 2 version as defined in BeamModulePlugin
+ spark_version = spark2_version
Review Comment:
As a side note - it's ambiguous to have two types of the naming style (underscore and camel case) in the same config file.
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -499,6 +499,10 @@ class BeamModulePlugin implements Plugin<Project> {
def arrow_version = "5.0.0"
def jmh_version = "1.34"
+ // Export Spark versions, so they are defined in a single place only
+ project.ext.spark2_version = spark2_version
Review Comment:
nit: just worth to mention that spark 2 is deprecated in Beam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1279209863
Run Spark StructuredStreaming ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1276312682
R: @aromanenko-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1276360312
# [Codecov](https://codecov.io/gh/apache/beam/pull/23603?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23603](https://codecov.io/gh/apache/beam/pull/23603?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7b6cc89) into [master](https://codecov.io/gh/apache/beam/commit/1c1ecb2a36dfb256a0e5eb7ff9ac5f84601b51e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c1ecb2) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #23603 +/- ##
==========================================
- Coverage 73.33% 73.33% -0.01%
==========================================
Files 719 719
Lines 95792 95792
==========================================
- Hits 70250 70248 -2
- Misses 24231 24233 +2
Partials 1311 1311
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.05% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/23603?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/23603/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | `92.44% <0.00%> (-0.65%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/23603/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.30% <0.00%> (-0.25%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/23603/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.24% <0.00%> (+0.16%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/23603/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `97.01% <0.00%> (+0.54%)` | :arrow_up: |
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/23603/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.02% <0.00%> (+0.74%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1279210095
Run Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1280659951
Run Java_Debezium_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged PR #23603:
URL: https://github.com/apache/beam/pull/23603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #23603:
URL: https://github.com/apache/beam/pull/23603#discussion_r998400384
##########
runners/spark/2/build.gradle:
##########
@@ -19,8 +19,8 @@
def basePath = '..'
/* All properties required for loading the Spark build script */
project.ext {
- // Set the version of all Spark-related dependencies here.
- spark_version = '2.4.8'
+ // Spark 2 version as defined in BeamModulePlugin
+ spark_version = spark2_version
Review Comment:
Not going to fix all inconsistencies at once ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a diff in pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on code in PR #23603:
URL: https://github.com/apache/beam/pull/23603#discussion_r998310688
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -499,6 +499,10 @@ class BeamModulePlugin implements Plugin<Project> {
def arrow_version = "5.0.0"
def jmh_version = "1.34"
+ // Export Spark versions, so they are defined in a single place only
+ project.ext.spark2_version = spark2_version
Review Comment:
nit: It'd be worth to mention that spark 2 is deprecated in Beam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #23603:
URL: https://github.com/apache/beam/pull/23603#discussion_r999635450
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -499,6 +499,10 @@ class BeamModulePlugin implements Plugin<Project> {
def arrow_version = "5.0.0"
def jmh_version = "1.34"
+ // Export Spark versions, so they are defined in a single place only
+ project.ext.spark2_version = spark2_version
Review Comment:
I'll add some comments as part of https://github.com/apache/beam/issues/23728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1279193962
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1280660608
Run Portable_Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23603: Keep Spark version in a single place only (BeamModulePlugin)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23603:
URL: https://github.com/apache/beam/pull/23603#issuecomment-1276379095
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org