You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/18 11:11:49 UTC
[GitHub] [beam] kamilwu opened a new pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
kamilwu opened a new pull request #12030:
URL: https://github.com/apache/beam/pull/12030
Because of a possible performance regression, the test executes too long and the Jenkins job is timing out.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12030:
URL: https://github.com/apache/beam/pull/12030#issuecomment-646003462
R: @udim @mxm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443419369
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
It could be related but it doesn't appear that any changes were made to the batch mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12030:
URL: https://github.com/apache/beam/pull/12030#issuecomment-645951105
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443060500
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
Also the comment should say "more than 4 hours"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443060213
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
As pointed out in the bug, there was sharp increase in `python_dataflow_batch_pardo_2_runtime` on May 7: http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=1584703436990&to=1592475836990&panelId=3&fullscreen
Perhaps this PR is related? https://github.com/apache/beam/pull/11558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu removed a comment on pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu removed a comment on pull request #12030:
URL: https://github.com/apache/beam/pull/12030#issuecomment-645951105
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dmvk commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443618694
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
@mxm Sorry, back from vacation.
Unfortunatelly I'm not really familiar with the portable runner. I guess that some of the recent "legacy" batch runner changes may have affected the portable runner too, but it's hard to say.
Can you `git bisect` to a commit that introduces the regression?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12030:
URL: https://github.com/apache/beam/pull/12030#issuecomment-645959828
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443569325
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
> Also the comment should say "more than 4 hours"
From https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/270/console:
```
16:21:33 INFO:apache_beam.runners.portability.portable_runner:Job state changed to RUNNING
19:48:20 Build timed out (after 240 minutes). Marking the build as aborted.
```
That gives us slightly more than 3 hours. The rest of time is spent on Pardo test with 10 iterations. I'll change the comment to "more than 3 hours"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r442212970
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 2 hours). Re-enable
Review comment:
@dmvk Since you are the batch expert, do you have any idea about this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12030:
URL: https://github.com/apache/beam/pull/12030#discussion_r443742088
##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [
environment_type : 'DOCKER',
]
],
- [
- title : 'ParDo Python Load test: 20M 100 byte records 200 times',
- test : 'apache_beam.testing.load_tests.pardo_test',
- runner : CommonTestProperties.Runner.PORTABLE,
- pipelineOptions: [
- job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
- project : 'apache-beam-testing',
- publish_to_big_query : true,
- metrics_dataset : datasetName,
- metrics_table : 'python_flink_batch_pardo_2',
- influx_measurement : 'python_batch_pardo_2',
- input_options : '\'{' +
- '"num_records": 20000000,' +
- '"key_size": 10,' +
- '"value_size": 90}\'',
- iterations : 200,
- number_of_counter_operations: 0,
- number_of_counters : 0,
- parallelism : 5,
- job_endpoint : 'localhost:8099',
- environment_type : 'DOCKER',
- ]
- ],
+// TODO(BEAM-10270): Takes too long time to execute (currently more than 3 hours). Re-enable
+// the test after its overhead is reduced.
Review comment:
Ok to disable for now to allow the remaining tests to run, but we should also investigate what caused the regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12030:
URL: https://github.com/apache/beam/pull/12030#issuecomment-645949404
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org