You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/08 14:29:31 UTC
[GitHub] [beam] kamilwu opened a new pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
kamilwu opened a new pull request #12195:
URL: https://github.com/apache/beam/pull/12195
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r457670348
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
Review comment:
We should move this suite to Py3 as we are actively discussing dropping Py2 support, and these suites will stop working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu merged pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu merged pull request #12195:
URL: https://github.com/apache/beam/pull/12195
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-661729360
Run Chicago Taxi on Dataflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-655579064
Run Chicago Taxi on Dataflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-661725899
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r456211925
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches("-PpipelineOptions=\"${LoadTestsBuilder.parseOptions(pipelineOptions)}\"")
}
}
}
+PostcommitJobBuilder.postCommitJob(
+ 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
+ 'Run Chicago Taxi on Dataflow',
+ 'Google Cloud Dataflow Runner Chicago Taxi Example',
Review comment:
It's nice to include trigger command in the message (easy to rerun if needed). How about:
Chicago Taxi Example on Dataflow (Run Chicago Taxi on Dataflow)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-656171851
Run Chicago Taxi on Dataflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-656202548
R: @tvalentyn
This effectively fixes https://issues.apache.org/jira/browse/BEAM-10421 as well. Look for "worker logs:" lines, there are links to Dataflow job details
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-655586849
Run Chicago Taxi on Dataflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-655556026
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-656163912
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r456214707
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
Review comment:
Can we switch this to Py3?
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches("-PpipelineOptions=\"${LoadTestsBuilder.parseOptions(pipelineOptions)}\"")
}
}
}
+PostcommitJobBuilder.postCommitJob(
+ 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
+ 'Run Chicago Taxi on Dataflow',
+ 'Google Cloud Dataflow Runner Chicago Taxi Example',
+ this
+) {
+ chicagoTaxiJob(delegate)
+}
+
CronJobBuilder.cronJob(
Review comment:
With CronJobBuilder and PostcommitJobBuilder, do we run these test as frequent as specified by Cron config or more frequently (as often as new commits come in) ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r456211925
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches("-PpipelineOptions=\"${LoadTestsBuilder.parseOptions(pipelineOptions)}\"")
}
}
}
+PostcommitJobBuilder.postCommitJob(
+ 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
+ 'Run Chicago Taxi on Dataflow',
+ 'Google Cloud Dataflow Runner Chicago Taxi Example',
Review comment:
It's nice to include trigger command in the message (easy to rerun if needed). How about:
Chicago Taxi Example on Dataflow ("Run Chicago Taxi on Dataflow")
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-658037656
@tvalentyn Would you mind taking a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-656151288
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-655572490
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-661791333
Thanks @tvalentyn. I agree we should move this suite to Py3, probably even as a part of https://issues.apache.org/jira/browse/BEAM-8892 (move Chicago Taxi to new Beam orchestrator developed by the TFX team). I will merge this PR to keep the test working until we move it to Py3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #12195: Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12195:
URL: https://github.com/apache/beam/pull/12195#issuecomment-655587469
Run Chicago Taxi on Dataflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r457552103
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
Review comment:
There's a jira for this: https://issues.apache.org/jira/browse/BEAM-9154. As far as I remember there was something that prevented us from switching to Py3. Not sure if it's still relevant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #12195: [BEAM-10422] Chicago Taxi example: set number of Dataflow workers to 5
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #12195:
URL: https://github.com/apache/beam/pull/12195#discussion_r457547061
##########
File path: .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
##########
@@ -19,46 +19,44 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
import CronJobBuilder
+import LoadTestsBuilder
-
-// This job runs the Chicago Taxi Example script on Dataflow
-PostcommitJobBuilder.postCommitJob(
- 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
- 'Run Chicago Taxi on Dataflow',
- 'Google Cloud Dataflow Runner Chicago Taxi Example',
- this
-) {
- description('Runs the Chicago Taxi Example on the Dataflow runner.')
+def chicagoTaxiJob = { scope ->
+ scope.description('Runs the Chicago Taxi Example on the Dataflow runner.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(scope)
+
+ def pipelineOptions = [
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
// Gradle goals for this job.
- steps {
+ scope.steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
tasks(':sdks:python:test-suites:dataflow:py2:chicagoTaxiExample')
switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches("-PpipelineOptions=\"${LoadTestsBuilder.parseOptions(pipelineOptions)}\"")
}
}
}
+PostcommitJobBuilder.postCommitJob(
+ 'beam_PostCommit_Python_Chicago_Taxi_Dataflow',
+ 'Run Chicago Taxi on Dataflow',
+ 'Google Cloud Dataflow Runner Chicago Taxi Example',
+ this
+) {
+ chicagoTaxiJob(delegate)
+}
+
CronJobBuilder.cronJob(
Review comment:
As frequent as specified by Cron only.
https://github.com/apache/beam/blob/0fc74c09f1b90c1b954b78973e83483daa51fd18/.test-infra/jenkins/CronJobBuilder.groovy#L47
The fourth parameter, `triggerOnCommit`, is set to False here, which means that the job is not triggered when new commits come in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org