You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "johnjcasey (via GitHub)" <gi...@apache.org> on 2023/04/06 16:41:59 UTC
[GitHub] [beam] johnjcasey opened a new pull request, #26153: add parallel number to gradle.properties
johnjcasey opened a new pull request, #26153:
URL: https://github.com/apache/beam/pull/26153
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1164721027
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/kafka/KafkaTestUtilities.groovy:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.gradle.kafka
+
+import org.gradle.api.Project
+import org.gradle.api.artifacts.ConfigurationContainer
+import org.gradle.api.tasks.testing.Test
+
+import javax.inject.Inject
+
+class KafkaTestUtilities {
+ abstract static class KafkaBatchIT extends Test {
+
+ @Inject
+ KafkaBatchIT(String delimited, String undelimited, Boolean sdfCompatible, ConfigurationContainer configurations, Project runningProject){
+ group = "Verification"
+ description = "Runs KafkaIO IT tests with Kafka clients API $delimited"
+ outputs.upToDateWhen { false }
+ testClassesDirs = runningProject.findProject(":sdks:java:io:kafka").sourceSets.test.output.classesDirs
+ classpath = configurations."kafkaVersion$undelimited" + runningProject.sourceSets.test.runtimeClasspath + runningProject.findProject(":sdks:java:io:kafka").sourceSets.test.runtimeClasspath
+
+ def pipelineOptions = [
+ '--sourceOptions={' +
+ '"numRecords": "1000",' +
+ '"keySizeBytes": "10",' +
+ '"valueSizeBytes": "90"' +
+ '}',
+ "--readTimeout=120",
+ "--kafkaTopic=beam",
+ "--withTestcontainers=true",
+ "--kafkaContainerVersion=5.5.2",
Review Comment:
Could we have these as variables that are then concatenated in the pipelineOptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26153: add parallel number to gradle.properties
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1503896797
## [Codecov](https://codecov.io/gh/apache/beam/pull/26153?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#26153](https://codecov.io/gh/apache/beam/pull/26153?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2035f2) into [master](https://codecov.io/gh/apache/beam/commit/a82a49bf163a27ef22a6ddbbe3db2eb960e39121?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a82a49b) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #26153 +/- ##
=======================================
Coverage 71.17% 71.17%
=======================================
Files 787 787
Lines 103293 103293
=======================================
Hits 73516 73516
Misses 28288 28288
Partials 1489 1489
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165687709
##########
sdks/java/io/kafka/kafka-01103/build.gradle:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.beam.gradle.kafka.KafkaTestUtilities
Review Comment:
Mostly, I wanted to avoid the same lines being repeated and you accomplished that with the shared properties file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165661339
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
I could add artificial dependencies between some of these tasks, to try and limit parallelism, but that would be a janky fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26153: add parallel number to gradle.properties
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505587938
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26153: add parallel number to gradle.properties
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1164495385
##########
sdks/java/io/kafka/kafka-100/build.gradle:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.beam.gradle.kafka.KafkaTestUtilities
Review Comment:
We can have a common.gradle in sdks/java/io/kafka and then build.gradle in kafka-{version} refers to it, like what flink runner support doing: https://github.com/apache/beam/blob/master/runners/flink/1.12/build.gradle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165683931
##########
sdks/java/io/kafka/kafka-01103/build.gradle:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.beam.gradle.kafka.KafkaTestUtilities
Review Comment:
I like the refactor here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505705980
run java precommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1502054625
Run Java_Kafka_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165658249
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
yeah, it looks like I can't configure parallelism differently for a subproject.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165579025
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
They are running fine on the Jenkins workers right now. I've tried to limit parallelism to just one sub-project, but I can only manage to configure the entire build to have a max number of workes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165686005
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
Ok - I can live without it at first. If we see these tests failing (at a higher than normal rate) with Jenkins infra-like issues, we should consider clawing this back though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505705643
run python_coverage precommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505706374
run java_gcp_io_direct precommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1501929644
Run Java_Kafka_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26153: add parallel number to gradle.properties
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505747592
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label java.
R: @damccorm for label build.
R: @ahmedabu98 for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165607732
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
I'll take another stab at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505705202
run python_runners precommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165498168
##########
sdks/java/io/kafka/kafka-01103/build.gradle:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.beam.gradle.kafka.KafkaTestUtilities
Review Comment:
Rather than having all of these in individual files, could we do this all in a loop from the base gradle file with something like https://github.com/apache/beam/blob/2325634e0314e28a2756f729ec75f5fd5d54bcbf/runners/spark/3/build.gradle#L53
##########
sdks/java/io/kafka/build.gradle:
##########
@@ -125,48 +116,24 @@ kafkaVersions.each {kv ->
}
}
-kafkaVersions.each {kv ->
- task "kafkaVersion${kv.key}BatchIT"(type: Test) {
- group = "Verification"
- description = "Runs KafkaIO IT tests with Kafka clients API $kv.value"
- outputs.upToDateWhen { false }
- testClassesDirs = sourceSets.test.output.classesDirs
- classpath = configurations."kafkaVersion${kv.key}" + sourceSets.test.runtimeClasspath
-
- def pipelineOptions = [
- '--sourceOptions={' +
- '"numRecords": "1000",' +
- '"keySizeBytes": "10",' +
- '"valueSizeBytes": "90"' +
- '}',
- "--readTimeout=120",
- "--kafkaTopic=beam",
- "--withTestcontainers=true",
- "--kafkaContainerVersion=5.5.2",
- ]
-
- systemProperty "beamTestPipelineOptions", groovy.json.JsonOutput.toJson(pipelineOptions)
- include '**/KafkaIOIT.class'
-
- filter {
- excludeTestsMatching "*InStreaming"
- if (!(kv.key in sdfKafkaVersions)) {
- excludeTestsMatching "*DynamicPartitions" //admin client create partitions does not exist in kafka 0.11.0.3 and kafka sdf does not appear to work for kafka versions <2.0.1
- excludeTestsMatching "*SDFResumesCorrectly" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*StopReadingFunction" //Kafka SDF does not work for kafka versions <2.0.1
- excludeTestsMatching "*WatermarkUpdateWithSparseMessages" //Kafka SDF does not work for kafka versions <2.0.1
- }
- }
- }
-}
-
+//Because this runs many integration jobs in parallel, each of which use a
+//container, it can fail locally due to performance limitations on a desktop.
+//To avoid this, use --max-workers=N, where N is less than half your CPUs.
+//4 is a good start for parallelism without overloading your computer.
Review Comment:
This is concerning to me, especially because Jenkins runs several jobs at the same time. If you run 2 of these jobs (or even worse, 4) on the same Jenkins machine, does the worker fall over?
I wonder if there's a version of this change that does some parallelization by default, but not this much (e.g. it spins up at most 4 containers at once)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1507282684
all failures are unrelated flakes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505707840
java precommit failure is a flake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505783935
That's a known issue with spanner permissions #26208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1505768353
GCP IO test failure is unrelated. We might have a quota issue on Spanner tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1506044095
All 4 failures appear to be unrelated flakes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #26153: add parallel number to gradle.properties
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #26153:
URL: https://github.com/apache/beam/pull/26153#issuecomment-1499343396
Run Java_Kafka_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey merged pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey merged PR #26153:
URL: https://github.com/apache/beam/pull/26153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #26153: Add Gradle sub-projects to enable parallel running of Kafka tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on code in PR #26153:
URL: https://github.com/apache/beam/pull/26153#discussion_r1165576831
##########
sdks/java/io/kafka/kafka-01103/build.gradle:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.beam.gradle.kafka.KafkaTestUtilities
Review Comment:
Nope. That is essentially what we had before, and we don't get parallelism with that because tasks within a project can't be run in parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org