You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/06 19:44:36 UTC
[GitHub] [beam] lukecwik opened a new pull request, #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
lukecwik opened a new pull request, #24928:
URL: https://github.com/apache/beam/pull/24928
This routes JCL/log4j/log4j2 messages to SLF4 which is then routed to JUL and finally over the Fn Logging API to the runner.
This change also prevents the usages of jul-to-slf4j as this would create an infinite logging loops in certain configurations and removes the existing patch work of exclusions.
Fixes #24927
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374725448
Run Java_JDBC_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] apilloud commented on a diff in pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
apilloud commented on code in PR #24928:
URL: https://github.com/apache/beam/pull/24928#discussion_r1063870868
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -486,6 +486,7 @@ class BeamModulePlugin implements Plugin<Project> {
def jsr305_version = "3.0.2"
def everit_json_version = "1.14.1"
def kafka_version = "2.4.1"
+ def log4j2_version = "2.19.0"
Review Comment:
Is there additional risk added by bumping the version? It was 2.17.1 before, not sure if this small version bump will be a problem. I think 2.17.1ish is in the Dataflow legacy worker container, will this conflict with that?
##########
sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle:
##########
@@ -28,15 +28,6 @@ ext.summary = "Common test classes for ElasticsearchIO"
def log4j_version = "2.17.1"
Review Comment:
nit: remove this?
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -573,6 +574,7 @@ class BeamModulePlugin implements Plugin<Project> {
commons_csv : "org.apache.commons:commons-csv:1.8",
commons_io : "commons-io:commons-io:2.7",
commons_lang3 : "org.apache.commons:commons-lang3:3.9",
+ commons_logging : "commons-logging:commons-logging:1.2",
Review Comment:
We explicitly remove commons_logging in flink with comments about jcl_over_slf4j. Do we need both?
https://github.com/apache/beam/blob/7d5d62e1ebf09eca0280bd7eea754a2223d0a416/runners/flink/job-server/flink_job_server.gradle#L75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374052648
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374050969
R: @apilloud @melaniedc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374224695
Java_PVR_Flink_Docker is one setup that covers the newly added test https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/2286/testReport/org.apache.beam.sdk/SdkHarnessEnvironmentTest/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik merged pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik merged PR #24928:
URL: https://github.com/apache/beam/pull/24928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a diff in pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on code in PR #24928:
URL: https://github.com/apache/beam/pull/24928#discussion_r1063889172
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -573,6 +574,7 @@ class BeamModulePlugin implements Plugin<Project> {
commons_csv : "org.apache.commons:commons-csv:1.8",
commons_io : "commons-io:commons-io:2.7",
commons_lang3 : "org.apache.commons:commons-lang3:3.9",
+ commons_logging : "commons-logging:commons-logging:1.2",
Review Comment:
commons_logging is only used in the sdks/java/core tests, the flink override still stands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a diff in pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on code in PR #24928:
URL: https://github.com/apache/beam/pull/24928#discussion_r1063889551
##########
sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle:
##########
@@ -28,15 +28,6 @@ ext.summary = "Common test classes for ElasticsearchIO"
def log4j_version = "2.17.1"
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374780390
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374087494
# [Codecov](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24928](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (96e3e39) into [master](https://codecov.io/gh/apache/beam/commit/07ba8f1c6c37cb526e5e9371256c92dbc56db870?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (07ba8f1) will **decrease** coverage by `0.08%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #24928 +/- ##
==========================================
- Coverage 73.16% 73.08% -0.09%
==========================================
Files 731 734 +3
Lines 97958 98095 +137
==========================================
+ Hits 71675 71691 +16
- Misses 24923 25044 +121
Partials 1360 1360
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `82.68% <ø> (-0.15%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../apache\_beam/runners/interactive/dataproc/types.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy90eXBlcy5weQ==) | `93.10% <0.00%> (-3.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.42% <0.00%> (ø)` | |
| [...chmarks/cloudml/cloudml\_benchmark\_constants\_lib.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2Nsb3VkbWwvY2xvdWRtbF9iZW5jaG1hcmtfY29uc3RhbnRzX2xpYi5weQ==) | `100.00% <0.00%> (ø)` | |
| [...m/testing/benchmarks/cloudml/pipelines/workflow.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2Nsb3VkbWwvcGlwZWxpbmVzL3dvcmtmbG93LnB5) | `6.94% <0.00%> (ø)` | |
| [...am/testing/benchmarks/cloudml/criteo\_tft/criteo.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2Nsb3VkbWwvY3JpdGVvX3RmdC9jcml0ZW8ucHk=) | `0.00% <0.00%> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.56% <0.00%> (+0.32%)` | :arrow_up: |
| [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `91.77% <0.00%> (+1.26%)` | :arrow_up: |
| [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/24928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `92.85% <0.00%> (+4.76%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a diff in pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on code in PR #24928:
URL: https://github.com/apache/beam/pull/24928#discussion_r1063889172
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -573,6 +574,7 @@ class BeamModulePlugin implements Plugin<Project> {
commons_csv : "org.apache.commons:commons-csv:1.8",
commons_io : "commons-io:commons-io:2.7",
commons_lang3 : "org.apache.commons:commons-lang3:3.9",
+ commons_logging : "commons-logging:commons-logging:1.2",
Review Comment:
commons_logging is only used in the sdks/java/core tests, the flink override is about making sure we use slf4j
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #24928:
URL: https://github.com/apache/beam/pull/24928#issuecomment-1374780449
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a diff in pull request #24928: [#24927] Ensure the SDK harness handles JCL/log4j/log4j2 messages.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on code in PR #24928:
URL: https://github.com/apache/beam/pull/24928#discussion_r1063890774
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -486,6 +486,7 @@ class BeamModulePlugin implements Plugin<Project> {
def jsr305_version = "3.0.2"
def everit_json_version = "1.14.1"
def kafka_version = "2.4.1"
+ def log4j2_version = "2.19.0"
Review Comment:
Dataflow legacy uses 2.17.2 which I swapped to here. Upgrading can happen in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org