You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/07 10:01:31 UTC
[GitHub] [beam] piotr-szuberski opened a new pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
piotr-szuberski opened a new pull request #13026:
URL: https://github.com/apache/beam/pull/13026
Update Kafka dependency from 1.0.0 to 2.6.0, Kafka clients from 1.0.0 to 2.4.1 and remove hack to use Kafka 2.1.0 in io/kafka
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
- Coverage 82.49% 82.48% -0.01%
==========================================
Files 455 455
Lines 54867 54867
==========================================
- Hits 45262 45259 -3
- Misses 9605 9608 +3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.07% <0.00%> (-0.27%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r523163048
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
It should do the same what kafkaVersion210 did. I've imported KafkaIO 2.25 jar and I can't see kafka-clients 2.1.0 in the dependencies.
I'm not sure how to import a jar to Gradle locally, I will try it out on Monday.
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
It should do the same what kafkaVersion210 did. I've imported KafkaIO 2.25 jar and I can't see kafka-clients 2.1.0 in the dependencies. So I don't think so.
I'm not sure how to import a jar to Gradle locally, I will try it out on Monday.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...c355a97](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727941678
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724694752
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...9eb14e7](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
- Coverage 82.49% 82.48% -0.01%
==========================================
Files 455 455
Lines 54867 54867
==========================================
- Hits 45262 45259 -3
- Misses 9605 9608 +3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.07% <0.00%> (-0.27%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...90ff6c4](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524248460
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
That's super useful! Thank you!
I can't see any Kafka clients in the created jar. Here is the output of :sdks:java:io:kafka:dependencies
```
compileClasspath - Compile classpath for source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
compileOnly - Compile only dependencies for source set 'main'. (n)
No dependencies
default - Configuration for default artifacts. (n)
No dependencies
implementation - Implementation only dependencies for source set 'main'. (n)
No dependencies
runtimeClasspath - Runtime classpath of source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
runtimeElements - Elements of runtime for main. (n)
No dependencies
runtimeOnly - Runtime only dependencies for source set 'main'. (n)
No dependencies
testAnnotationProcessor - Annotation processors and their dependencies for source set 'test'.
No dependencies
testCompileClasspath - Compile classpath for source set 'test'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727145508
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-728249571
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724204377
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724696322
@aromanenko-dev @iemejia Kafka previous versions tests (including IT) are now running in Java Postcommit - is this solution acceptable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [d8f9cd9...321315c](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524364352
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
Kafka performance tests have been failing (timeout) since https://github.com/apache/beam/pull/13282 merge.
I guess this PR should wait until it's fixed?
https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r523141046
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
This task is still there at
```
kafkaVersions.each {kv ->
task "kafkaVersion${kv.key}Test"(type: Test)
[...]
```
Though I used 2.1.1 version - should we leave 2.1.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727145533
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727145508
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...195e60d](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724694752
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [beee718...d15f3d0](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
- Coverage 82.49% 82.48% -0.01%
==========================================
Files 455 455
Lines 54867 54867
==========================================
- Hits 45262 45259 -3
- Misses 9605 9608 +3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.07% <0.00%> (-0.27%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733782983
@aromanenko-dev Ok I'm out of ideas.
Jenkins takes `**/build/test-results/**.xml` files to print the result of executed tests.
All of the kafka versions tests are the same (KafkaIOIT, KafkaIOTest), so they override each other in the test result dashboard.
I tried to rename these xml files but it looks like Jenkins cares about the info inside them, thus renaming has no effect.
There seems to be no simple way to just print the version for each test execution.
Maybe it's sufficient to just know that some Kafka version is broken and run the tests locally? They don't take much time and everything is printed nicely when run on a local machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [WIP][BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [405204c...5885e43](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727947192
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-728963465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734359531
@chamikaramj @boyuanzz I guess you can be interested in this PR too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727942075
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [405204c...5648bc0](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727603657
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524267637
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
I'm not sure as well but I guess so. I'm having a difficulty with the recently added Kafka streaming performance tests so I'm changing this PR to WIP.
If I understand correctly these streaming tests can't be run on DirectRunner? They never stop on both direct and flink even with `readTimeout` set to 1min
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.51% 82.55% +0.04%
==========================================
Files 455 455
Lines 54867 55143 +276
==========================================
+ Hits 45272 45526 +254
- Misses 9595 9617 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (-0.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.34%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [2f2ffda...f7d06ba](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707866031
This is quite significant change since Beam tended to support most of the Kafka client versions almost from the beginning. I believe that there is still a decent amount of users that uses Beam with quite old Kafka client versions [1]. So, before making a final decision, I propose to do a survey on user@beam.apache.org and see the results.
[1] https://lists.apache.org/thread.html/39ca4354f1e4a95211c90146a6acc123d17afc0a4cbec08ac7814845%40%3Cuser.beam.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733782983
@aromanenko-dev Ok I'm out of ideas.
Jenkins takes `**/build/test-results/**.xml` files to print the result of executed tests.
All of the kafka versions tests are the same (KafkaIOIT, KafkaIOTest), so they override each other in the test result dashboard.
I tried to rename these xml files but it looks like Jenkins cares about the info inside them.
So there seems to be no simple way to just print the version for each test execution.
Maybe it's sufficient to just know that some Kafka version is broken and run the tests locally? They don't take much time and everything is printed nicely when run on a local machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524248460
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
That's super useful! Thank you!
I can't see any Kafka clients in the created jar. from a newly created project that uses the local repo:
```
repositories {
mavenLocal()
}
dependencies {
compile group: 'org.apache.beam', name: 'beam-sdks-java-io-kafka', version: '2.27.0-SNAPSHOT'
testCompile group: 'org.apache.beam', name: 'beam-sdks-java-io-kafka', version: '2.27.0-SNAPSHOT'
}
```
Here is the output of `dependencies` task
```
compileClasspath - Compile classpath for source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
compileOnly - Compile only dependencies for source set 'main'. (n)
No dependencies
default - Configuration for default artifacts. (n)
No dependencies
implementation - Implementation only dependencies for source set 'main'. (n)
No dependencies
runtimeClasspath - Runtime classpath of source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
runtimeElements - Elements of runtime for main. (n)
No dependencies
runtimeOnly - Runtime only dependencies for source set 'main'. (n)
No dependencies
testAnnotationProcessor - Annotation processors and their dependencies for source set 'test'.
No dependencies
testCompileClasspath - Compile classpath for source set 'test'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733662099
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sap1ens commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
sap1ens commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707468704
Just two cents about Kafka compatibility, it's been working extremely well since 0.10.2 ([this is a good overview about the design](https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/)), so I believe the proposed upgrade here should be very straightforward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727941678
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-727942075
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-716643515
> @piotr-szuberski What do you mean by "it was not accepted"? I can see you email "[DISCUSS] Update Kafka dependencies in Beam Java SDK" from Oct 22nd on user@
I just wasn't subscribed to user@ mailing list before and the thread was disappearing without any message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-712247471
> This is quite significant change since Beam tended to support most of the Kafka client versions almost from the beginning. I believe that there is still a decent amount of users that uses Beam with quite old Kafka client versions [1]. So, before making a final decision, I propose to do a survey on [user@beam.apache.org](mailto:user@beam.apache.org) and see the results.
I've created a survey on the users mailing list but it was probably not accepted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.51% 82.55% +0.04%
==========================================
Files 455 455
Lines 54867 55143 +276
==========================================
+ Hits 45272 45526 +254
- Misses 9595 9617 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (-0.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.34%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-705381413
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-716601689
@piotr-szuberski What do you mean by "it was not accepted"? I can see you email "[DISCUSS] Update Kafka dependencies in Beam Java SDK" from Oct 22nd on user@
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [beee718...84fbee7](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...7a4660b](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...bd392dc](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-735645554
@aromanenko-dev From my side it's ready to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734420606
> @piotr-szuberski It seems a bit odd but I don't think it's a blocker for this PR. In any case, we have a Kafka version in the gradle task name. So if it fails then the corresponding task will fail too, right? And we can see that in Jenkins job logs, like [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/511/consoleText).
Yes, exactly like [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/506/console):
```
00:18:25 Execution failed for task ':sdks:java:io:kafka:kafkaVersion090BatchIT'.
```
>
> So, in general it LGTM. Do you have something to add to this PR? Otherwise, please undraft it to make it possible to merge.
Great! I've made some cleanup and removed the xml renaming part. When it's green then it's ready for merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...8b2a7f3](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...d1ff41d](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524226883
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
Could you try to build and publish KafkaIO jar with this command:
```
./gradlew -Ppublishing -PdistMgmtSnapshotsUrl=/path/to/.m2/repository/ -p sdks/java/io/kafka publishToMavenLocal
```
and check out which Kafka clients deps it comtains?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524267637
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
I'm not sure as well but I guess so. I'm having a difficulty with the recently added Kafka streaming performance tests so I'm changing this PR to WIP.
If I understand correctly these streaming tests can't be run on DirectRunner? They never stop even with `readTimeout` set to 1min
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-737083007
> @piotr-szuberski @aromanenko-dev @iemejia
>
> This broke the Spark postcommit: [BEAM-11375](https://issues.apache.org/jira/browse/BEAM-11375)
>
> It looks like the cause is the mismatch between `kafka_2.11` (2.4.1) and `kafka-clients` (2.6.0). Kafka 2.5.0 deprecated support for Scala 2.11, so I'm guessing we can't bring it up to date until upgrading Spark to Scala 2.12?
Sorry for that, I didn't take it into account. I'll try to either update the scala version of Kafka dependency (not sure if it's that easy if Spark doesn't support Scala 2.12) or simply downgrade kafka client to 2.4.1
Scala update PR: https://github.com/apache/beam/pull/13458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] methodmissing commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
methodmissing commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707050839
I have a branch in progress to run the `io/kafka` tests with more kafka-client versions in https://github.com/apache/beam/compare/master...methodmissing:kafkaio-test-more-versions but haven't had a window to converge on that, but can prioritise otherwise.
Related, but forcing a newer kafka-clients version, I think that'll mostly invalidate `ConsumerSpEL` and `ProducerSpEL` as well.
@lukecwik thoughts on the implications of bumping default kafka-clients up from 1.0.0 to 2.4.1 in 1 fell swoop? It's runtime pluggable, at Shopify we already use 2.4.1 for a long time and moving towards 2.6.0 for Beam but I worry about backwards compatibility. Also admittedly biased because https://github.com/apache/beam/pull/12794 would have been way less complex to implement and users of `1.0.0` in the wild is setting themselves up for other problems moving forward too 🤷♂️ :smile:
cc @sap1ens @vvuibert
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729804996
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707223491
> I have a branch in progress to run the `io/kafka` tests with more kafka-client versions in [master...methodmissing:kafkaio-test-more-versions](https://github.com/apache/beam/compare/master...methodmissing:kafkaio-test-more-versions) but haven't had a window to converge on that, but can prioritise otherwise.
>
> Related, but forcing a newer kafka-clients version, I think that'll mostly invalidate `ConsumerSpEL` and `ProducerSpEL` as well.
>
> @lukecwik thoughts on the implications of bumping default kafka-clients up from 1.0.0 to 2.4.1 in 1 fell swoop? It's runtime pluggable, at Shopify we already use 2.4.1 for a long time and moving towards 2.6.0 for Beam but I worry about backwards compatibility. Also admittedly biased because #12794 would have been way less complex to implement and users of `1.0.0` in the wild is setting themselves up for other problems moving forward too 🤷♂️ 😄
>
> cc @sap1ens @vvuibert
The purpose of supporting multiple versions is to have greater flexibility for users so the change in #12794 would have been equally complex I believe.
Typically I would lean to allowing this because we typically do perform dependency updates regularly but we have had issues in the past with some libraries like Avro and since we were already doing something special here likely means that it is important to the user community. Its best to ask on dev@beam.apache.org for feedback on this change (@aromanenko-dev and @iemejia should know more about this).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733782983
@aromanenko-dev Ok I'm out of ideas.
Jenkins takes `**/build/test-results/**.xml` files to print the result of executed tests.
All of the kafka versions tests are the same (KafkaIOIT, KafkaIOTest), so they override each other in the test result dashboard.
I tried to rename these xml files but it looks like Jenkins cares about the info inside them, thus renaming has no effect.
There seems to be no simple way to just print the version for each test execution.
Maybe it's sufficient to just know that some Kafka version is broken and run the tests locally? They don't take much time and everything is printed nicely when run on a local machine.
Some dynamically generated test cases names would solve the problem but I can't find anything in junit4 that would fit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-726147085
@piotr-szuberski Thanks, sorry for delay with answer, I'll take a look asap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733717463
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [WIP][BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724055215
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724061728
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733007210
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-716643515
> @piotr-szuberski What do you mean by "it was not accepted"? I can see you email "[DISCUSS] Update Kafka dependencies in Beam Java SDK" from Oct 22nd on user@
I just wasn't subscribed to user@ mailing list before and the thread was disappearing after creation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729022328
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724055347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [WIP][BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724055347
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733008640
> @piotr-szuberski Actually, I proposed to print out the Kafka client version because now [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/507/testReport/org.apache.beam.sdk.io.kafka/KafkaIOIT/) we can see the list of the executed tests but if some of them failed, we can't say for sure which Kafka version were used. Do I miss something?
You're right about it, I've tried to print some logs from the tests but they are not visible on Jenkins.
I'll try to add some gradle-level error logs. It looks like printing "info" logs is not sufficient because postcommit task doesn't have '--info' switch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-705381134
R: @methodmissing Could I ask you for review? I removed the code enforcing kafka clients 2.1.0 version since it's now 2.6.0 by default. I'm not sure whether I'm not missing something though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734726083
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724696322
@aromanenko-dev @iemejia I've updated this PR. Kafka previous versions tests (including IT) are now running in Java Postcommit - is this solution acceptable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r525273614
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #13026:
URL: https://github.com/apache/beam/pull/13026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...e53f682](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724627274
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524484370
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
Looking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729022828
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724694243
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729972853
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [15cdc43...35958bd](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734357293
@piotr-szuberski It seems a bit odd but I don't think it's a blocker for this PR. In any case, we have a Kafka version in the gradle task name. So if it fails then the corresponding task will fail too, right? And we can see that in Jenkins job logs, like [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/511/consoleText).
So, in general it LGTM. Do you have something to add to this PR? Otherwise, please undraft it to make it possible to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-732771358
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-705381134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734419234
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r525263070
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
Sorry, I missed the request for logging the Kafka version. I'm working on it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733099644
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-705381250
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-737083007
> @piotr-szuberski @aromanenko-dev @iemejia
>
> This broke the Spark postcommit: [BEAM-11375](https://issues.apache.org/jira/browse/BEAM-11375)
>
> It looks like the cause is the mismatch between `kafka_2.11` (2.4.1) and `kafka-clients` (2.6.0). Kafka 2.5.0 deprecated support for Scala 2.11, so I'm guessing we can't bring it up to date until upgrading Spark to Scala 2.12?
Sorry for that, I didn't take it into account. I'll try to either update the scala version of Kafka dependency (not sure if it's that easy if Spark doesn't support Scala 2.12) or simply downgrade kafka client to 2.4.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r523015223
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
Will it require to include all these different Kafka client version jars into regular build?
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
I think this task was used to run KafkaIO unit tests against specific client version. It's not a case now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-732022234
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524416279
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
@boyuanzz Could you take a look on https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1563/ ? After rebase with latest KafkaIOIT changes, It fails because of timeout.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524491777
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
It seems like the sdk docker image is cleaned up incorrectly before test finishes. I can have a quick fix for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524228547
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
I think it should fine since 2.1.0 was used as a minimum version where headers were added (if I'm not mistaken).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r523015223
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
Will it require to include all these different Kafka client version jars into regular build as deps?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724298118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724204377
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729801867
For some reason the printed versions are correct when run locally and on Jenkins it's always `INFO: Running test for Kafka version 0.11.0.3.`. I'll add some better asserion to IT tests and try to figure out what's going on. Maybe Gradle runs everything in parallel and overwrites the previous kafka clients jar versions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-737083007
> @piotr-szuberski @aromanenko-dev @iemejia
>
> This broke the Spark postcommit: [BEAM-11375](https://issues.apache.org/jira/browse/BEAM-11375)
>
> It looks like the cause is the mismatch between `kafka_2.11` (2.4.1) and `kafka-clients` (2.6.0). Kafka 2.5.0 deprecated support for Scala 2.11, so I'm guessing we can't bring it up to date until upgrading Spark to Scala 2.12?
Sorry for that, I didn't take it into account. I'll try to either update the scala version of Kafka dependency or downgrade kafka clients to 2.4.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-732199535
@aromanenko-dev Ok, despite printing all the time the same version it looks like it's working. I tried to put Kafka 0.9 among the Kafka versions and it indeed failed (still printing 0.11.3).
So I removed the version printing and added a new test to the IT that verifies that the version from AppInfoParser is the same as in the kafka versions map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.51% 82.55% +0.04%
==========================================
Files 455 455
Lines 54867 55143 +276
==========================================
+ Hits 45272 45526 +254
- Misses 9595 9617 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (-0.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.34%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...8a7d821](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] je-ik commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707709558
I'll just metion one issue I ran into, `poll` can now throw `RebalanceInProgressException` that must be handled: https://kafka.apache.org/26/javadoc/org/apache/kafka/common/errors/RebalanceInProgressException.html.
This applies only to cases when consumer `group id` used, we don't use it by default, but when user specifies it, the handling probably should be added. This change was introduced in KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [beee718...dbba3aa](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
- Coverage 82.49% 82.48% -0.01%
==========================================
Files 455 455
Lines 54867 54867
==========================================
- Hits 45262 45259 -3
- Misses 9605 9608 +3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.07% <0.00%> (-0.27%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524248460
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
Review comment:
That's super useful! Thank you!
I can't see any Kafka clients in the created jar. I created a new project that uses the local repo:
```
repositories {
mavenLocal()
}
dependencies {
compile group: 'org.apache.beam', name: 'beam-sdks-java-io-kafka', version: '2.27.0-SNAPSHOT'
testCompile group: 'org.apache.beam', name: 'beam-sdks-java-io-kafka', version: '2.27.0-SNAPSHOT'
}
```
Here is the output of `dependencies` task
```
compileClasspath - Compile classpath for source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
compileOnly - Compile only dependencies for source set 'main'. (n)
No dependencies
default - Configuration for default artifacts. (n)
No dependencies
implementation - Implementation only dependencies for source set 'main'. (n)
No dependencies
runtimeClasspath - Runtime classpath of source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
runtimeElements - Elements of runtime for main. (n)
No dependencies
runtimeOnly - Runtime only dependencies for source set 'main'. (n)
No dependencies
testAnnotationProcessor - Annotation processors and their dependencies for source set 'test'.
No dependencies
testCompileClasspath - Compile classpath for source set 'test'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 -> 2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [WIP][BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-719321224
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-705381413
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-732982763
@piotr-szuberski Actually, I proposed to print out the Kafka client version because now [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/507/testReport/org.apache.beam.sdk.io.kafka/KafkaIOIT/) we can see the list of the executed tests but if some of them failed, we can't say for sure which Kafka version were used. Do I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-728963465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-729501229
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r523141046
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
This task is still there at
```
kafkaVersions.each {kv ->
task "kafkaVersion${kv.key}Test"(type: Test)
[...]
```
Though I used 2.1.1 version - is 2.1.0 special for some reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-734357293
@piotr-szuberski It seems a bit odd but I don't think it's a blocker for this PR. In any case, we have a Kafka version in the gradle task name. So if it fails then the corresponding task will fail too, right? And we can see that in Jenkins job logs, like [here](https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/511/consoleText).
So, in general it LGTM. Do you have something to add to this PR? Otherwise, we can merge that I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
- Coverage 82.49% 82.48% -0.01%
==========================================
Files 455 455
Lines 54867 54867
==========================================
- Hits 45262 45259 -3
- Misses 9605 9608 +3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.07% <0.00%> (-0.27%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.03% <0.00%> (+0.28%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524517625
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
So the problem is introduced by https://github.com/apache/beam/pull/13283/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] je-ik edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
je-ik edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-707709558
I'll just metion one issue I ran into, `poll` can now throw `RebalanceInProgressException` that must be handled: https://kafka.apache.org/26/javadoc/org/apache/kafka/common/errors/RebalanceInProgressException.html.
This applies only to cases when consumer `group.id` used, we don't use it by default, but when user specifies it, the handling probably should be added. This change was introduced in KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-737083007
> @piotr-szuberski @aromanenko-dev @iemejia
>
> This broke the Spark postcommit: [BEAM-11375](https://issues.apache.org/jira/browse/BEAM-11375)
>
> It looks like the cause is the mismatch between `kafka_2.11` (2.4.1) and `kafka-clients` (2.6.0). Kafka 2.5.0 deprecated support for Scala 2.11, so I'm guessing we can't bring it up to date until upgrading Spark to Scala 2.12?
Sorry for that, I didn't take it into account. I'll try to either update the scala version of Kafka dependency (not sure if it's that easy if Spark doesn't support Scala 2.12) or simply downgrade kafka client to 2.4.1
Reversion PR: https://github.com/apache/beam/pull/13458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) (000ac07) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.48% 82.55% +0.07%
==========================================
Files 455 455
Lines 54876 55143 +267
==========================================
+ Hits 45266 45526 +260
- Misses 9610 9617 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.45% <0.00%> (-0.36%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <0.00%> (-0.08%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (ø)` | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.24% <0.00%> (+1.75%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.90%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [beee718...3309f67](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-712247471
> This is quite significant change since Beam tended to support most of the Kafka client versions almost from the beginning. I believe that there is still a decent amount of users that uses Beam with quite old Kafka client versions [1]. So, before making a final decision, I propose to do a survey on [user@beam.apache.org](mailto:user@beam.apache.org) and see the results.
I've created a survey on the users mailing list but probably it was not accepted for some reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13026: [BEAM-8639][BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-704839820
# [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=h1) Report
> Merging [#13026](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13026 +/- ##
==========================================
+ Coverage 82.51% 82.55% +0.04%
==========================================
Files 455 455
Lines 54867 55143 +276
==========================================
+ Hits 45272 45526 +254
- Misses 9595 9617 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.32% <0.00%> (-1.20%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.57% <0.00%> (-0.18%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.34% <0.00%> (-0.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.63% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/13026/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `90.69% <0.00%> (+2.34%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=footer). Last update [6bf56f9...000ac07](https://codecov.io/gh/apache/beam/pull/13026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-724696322
@aromanenko-dev @iemejia I updated this PR. Kafka previous versions tests (including IT) are now running in Java Postcommit - is this solution acceptable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-736792106
@piotr-szuberski @aromanenko-dev @iemejia
This broke the Spark postcommit: BEAM-11375
It looks like the cause is the mismatch between `kafka_2.11` (2.4.1) and `kafka-clients` (2.6.0). Kafka 2.5.0 deprecated support for Scala 2.11, so I'm guessing we can't bring it up to date until upgrading Spark to Scala 2.12?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #13026:
URL: https://github.com/apache/beam/pull/13026#issuecomment-733662099
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13026: [WIP] [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r525140909
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key" "org.apache.kafka:kafka-clients:$it.value"}
}
-configurations.kafkaVersion210 {
- resolutionStrategy {
- force "org.apache.kafka:kafka-clients:2.1.0"
+kafkaVersions.each { kv ->
+ configurations."kafkaVersion$kv.key" {
+ resolutionStrategy {
+ force "org.apache.kafka:kafka-clients:$kv.value"
+ }
}
}
-task kafkaVersion210Test(type: Test) {
Review comment:
@boyuanzz Thanks! I guess it's should fixed now (#13356 ), right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org