You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/11/07 01:37:17 UTC
[GitHub] [beam] boyuanzz opened a new pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
boyuanzz opened a new pull request #13282:
URL: https://github.com/apache/beam/pull/13282
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724223455
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724200309
> Thanks! I'll take a look asap
Thanks! I'm curious is it intent that the original KafkaIO performance test runs in batch mode? That seems wired to me since the major usage for Kafka is in streaming and Dataflow has different execution path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-723373292
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724217047
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725780350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726399639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r522169736
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
Imho, it's fine if it's just temporary but if it's going to be exposed to user or even for the sake of clarity, I'd prefer to have additional string value, like `dataflow_v2` for instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r521635255
##########
File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -1442,7 +1447,7 @@ class BeamModulePlugin implements Plugin<Project> {
JavaPerformanceTestConfiguration configuration = it ? it as JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
// Task for running integration tests
- project.task('integrationTest', type: Test) {
+ def itTask = project.task('integrationTest', type: Test) {
Review comment:
This file defines several common project configurations, which can be applied to any subproject.
Comparing to the solution we have for dataflow build, one benefit of using such configuration to create IT is that we can have one centralized place for all IO to set up ITs instead of having them one by one and it's easier to use one test configuration to invoke tests on different runners. Taking this KafkaIO IT as an example, if one day Flink is interested in this suite, Flink can just use the same configuration in https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726427774
All tests are passing. Please let me know if you have any other concerns. @aromanenko-dev
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726145044
@boyuanzz Could be `Python PreCommit` failed related to this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r522134032
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
Hmm, maybe it's not related to this PR, but the value `use_runner_v2` doesn't reflect that it's related only to Dataflow, I'd assume mostly Runner v2 API looking on this. Do you know if it's already defined term?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725251852
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev edited a comment on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726145787
nit: Also, please add `[BEAM-11172] ` prefix to commit message and rebase your branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r520677337
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
What actually "runner v2" is?
##########
File path: sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
##########
@@ -115,6 +122,43 @@ public static void setup() throws IOException {
.get();
}
+ @Test
+ public void testKafkaIOWithRunnerV2() throws IOException {
+ writePipeline
+ .apply("Generate records", Read.from(new SyntheticBoundedSource(sourceOptions)))
+ .apply("Measure write time", ParDo.of(new TimeMonitor<>(NAMESPACE, WRITE_TIME_METRIC_NAME)))
+ .apply("Write to Kafka", writeToKafka());
+
+ readPipeline.getOptions().as(Options.class).setStreaming(true);
+ PCollection<Integer> elementCount =
+ readPipeline
+ .apply("Read from Runner V2 Kafka", readFromKafka())
+ .apply(
+ "Measure read time", ParDo.of(new TimeMonitor<>(NAMESPACE, READ_TIME_METRIC_NAME)))
+ .apply("Map records to strings", MapElements.via(new MapKafkaRecordsToStrings()))
+ .apply(
+ "Keyed by empty key",
+ MapElements.into(new TypeDescriptor<KV<byte[], String>>() {})
+ .via(element -> KV.of(new byte[0], element)))
+ .apply(
+ "Counting elements", ParDo.of(new CountingElementFn(options.getNumberOfRecords())));
Review comment:
Why not to use metrics for counting?
##########
File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java
##########
@@ -407,4 +408,13 @@ public double getTotalSize(double numRecords) {
return avgRecordSize.get() * numRecords / (1 + avgRecordGap.get());
}
}
+
+ private static Instant ensureTimestampWithinBounds(Instant timestamp) {
Review comment:
Why do we need this function? Is it possible to have a timestamp out of window bounds?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726367753
> @boyuanzz Could be `Python PreCommit` failed related to this change?
No, the failure is not related. It should be fixed when https://github.com/apache/beam/pull/13321 is checked in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev edited a comment on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726145787
nit: Also, please add "[BEAM-11172] " prefix to commit message and rebase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725202866
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev edited a comment on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724678000
> Thanks! I'm curious is it intent that the original KafkaIO performance test runs in batch mode? That seems wired to me since the major usage for Kafka is in streaming and Dataflow has different execution path for batch and streaming.
Afaict, `KafkaIOIT` was initially mostly an integration test and it was done similar to other ITs, developed for bounded IOs. The main idea of such tests is to write a limited number of pre-defined records (for which we already calculated the right hash sum) and then read them with IO's `Read` transform and compare the hashes. We can probably do similar for unbounded source just with significant amount of waiting time for read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r521603774
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,60 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
Review comment:
Maybe including the word 'Dataflow' would be informative for these configs.
##########
File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -233,6 +233,8 @@ class BeamModulePlugin implements Plugin<Project> {
// Required. Pipeline options to be used by the tested pipeline.
String integrationTestPipelineOptions = System.getProperty('integrationTestPipelineOptions')
+
+ String experiment = System.getProperty('experiment', '')
Review comment:
Adding an 'experiment' flag here could be confused with the pipelineOptions experiment flag. Maybe instead of this, it would be better to inspect the pipelineOptions instead?
##########
File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -1442,7 +1447,7 @@ class BeamModulePlugin implements Plugin<Project> {
JavaPerformanceTestConfiguration configuration = it ? it as JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
// Task for running integration tests
- project.task('integrationTest', type: Test) {
+ def itTask = project.task('integrationTest', type: Test) {
Review comment:
I'm not very familiar with this file, but it seems like we're now configuring runnerv2 tasks (e.g. cleanup tasks, building container images) based on various settings (e.g. pipelineOptions, the gradle 'experiment' flag) here and also in various build.gradle files.
Should we be picking one over the other? For example, the PR you're reviewing of mine that contains IT tests for examples, configures the tests in the build.gradle file. It seems strikingly similar to what is going on here though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz edited a comment on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz edited a comment on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724200309
> Thanks! I'll take a look asap
Thanks! I'm curious is it intent that the original KafkaIO performance test runs in batch mode? That seems wired to me since the major usage for Kafka is in streaming and Dataflow has different execution path for batch and streaming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r522329137
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
For testing, we need this string to decided whether to build the sdk container. In the latest change, I have removed this argument and checked pipeline options string only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r520984675
##########
File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java
##########
@@ -407,4 +408,13 @@ public double getTotalSize(double numRecords) {
return avgRecordSize.get() * numRecords / (1 + avgRecordGap.get());
}
}
+
+ private static Instant ensureTimestampWithinBounds(Instant timestamp) {
Review comment:
The MIN_TIMESTAMP produced in Dataflow is smaller than `BoundedWindow.MIN_TIMESTAMP`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725807114
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724678000
> Thanks! I'm curious is it intent that the original KafkaIO performance test runs in batch mode? That seems wired to me since the major usage for Kafka is in streaming and Dataflow has different execution path for batch and streaming.
Afaict, `KafkaIOIT` was initially mostly an integration test and it was done similar to other ITs, developed for bounded IOs. The main idea of such tests are to write a limited number of pre-defined records (for which we already calculated the right hash) and then read it with IO's `Read` transform and compare the hashes. We can probably do similar for unbounded source just with significant amount of waiting time for read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725726772
> Thanks, looks good to me now.
Thanks for your help! For next step, we could try to add runner_v2 to other IO performance suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r521680759
##########
File path: sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
##########
@@ -115,16 +116,55 @@ public static void setup() throws IOException {
.get();
}
+ @Test
+ public void testKafkaIOWithRunnerV2() throws IOException {
Review comment:
That's correct. I'll do the renaming. Thanks for catching that!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724308356
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725696145
One successful run: https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1539/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725083681
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725640633
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725190779
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725723048
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r520978104
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
`runner v2` is a new version of Dataflow. We are under migration from old Dataflow runner to this new runner. We(dataflow) are more interested in making sure SDF and KafkaIO work well on the new runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r521647285
##########
File path: sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
##########
@@ -115,16 +116,55 @@ public static void setup() throws IOException {
.get();
}
+ @Test
+ public void testKafkaIOWithRunnerV2() throws IOException {
Review comment:
Instead of naming the test after the runner, is it possible to name the test after the behavior that is being tested and use the configuration to ensure it runs on the appropriate runners? Is there a reason to have both this test and `testKafkaReadsAndWritesProperly`?
If so, it would be good to understand the difference through the test naming. From what I can tell the material difference is that this test uses a streaming pipeline to read where `testKafkaReadsAndWritesProperly` uses a batch one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725598255
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725720064
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-726145787
nit: Also, please add "[BEAM-11172] " prefix to commit message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725081840
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725210904
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725637391
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725599676
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724150801
Thanks! I'll take a look asap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725593409
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725157411
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-723386118
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725219513
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-723386707
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725208671
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r522161357
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments : 'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
That's a good point. The Dataflow backend expects that exact string but for Beam users `dataflow_v2` makes more sense. It is just an experiment and temporary, though. Hopefully most users will never need to set it. We could add the additional string and just swap it out when we submit the job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r522185225
##########
File path: sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
##########
@@ -116,15 +117,54 @@ public static void setup() throws IOException {
}
@Test
- public void testKafkaIOReadsAndWritesCorrectly() throws IOException {
+ public void testKafkaIOReadsAndWritesCorrectlyInStreaming() throws IOException {
+ // Use batch pipeline to write records.
+ writePipeline
+ .apply("Generate records", Read.from(new SyntheticBoundedSource(sourceOptions)))
+ .apply("Measure write time", ParDo.of(new TimeMonitor<>(NAMESPACE, WRITE_TIME_METRIC_NAME)))
+ .apply("Write to Kafka", writeToKafka());
+
+ // Use streaming pipeline to read Kafka records.
+ readPipeline.getOptions().as(Options.class).setStreaming(true);
+ readPipeline
+ .apply("Read from Runner V2 Kafka", readFromKafka())
Review comment:
nit: Please, remove "Runner V2"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725225778
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #13282:
URL: https://github.com/apache/beam/pull/13282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725091951
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-725755236
@aromanenko-dev ptal : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-723374147
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-724310859
Run Java KafkaIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13282:
URL: https://github.com/apache/beam/pull/13282#issuecomment-723370448
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #13282: [BEAM-11172] Enable KafkaIO performance test for runner v2 with SDF.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r520985187
##########
File path: sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
##########
@@ -115,6 +122,43 @@ public static void setup() throws IOException {
.get();
}
+ @Test
+ public void testKafkaIOWithRunnerV2() throws IOException {
+ writePipeline
+ .apply("Generate records", Read.from(new SyntheticBoundedSource(sourceOptions)))
+ .apply("Measure write time", ParDo.of(new TimeMonitor<>(NAMESPACE, WRITE_TIME_METRIC_NAME)))
+ .apply("Write to Kafka", writeToKafka());
+
+ readPipeline.getOptions().as(Options.class).setStreaming(true);
+ PCollection<Integer> elementCount =
+ readPipeline
+ .apply("Read from Runner V2 Kafka", readFromKafka())
+ .apply(
+ "Measure read time", ParDo.of(new TimeMonitor<>(NAMESPACE, READ_TIME_METRIC_NAME)))
+ .apply("Map records to strings", MapElements.via(new MapKafkaRecordsToStrings()))
+ .apply(
+ "Keyed by empty key",
+ MapElements.into(new TypeDescriptor<KV<byte[], String>>() {})
+ .via(element -> KV.of(new byte[0], element)))
+ .apply(
+ "Counting elements", ParDo.of(new CountingElementFn(options.getNumberOfRecords())));
Review comment:
Yes, counter works as well. I think we can use existing element counter as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org