You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/05/25 11:54:31 UTC
[GitHub] [beam] damccorm opened a new pull request, #17752: [BEAM-14509] Add several flags to dataflow runner
damccorm opened a new pull request, #17752:
URL: https://github.com/apache/beam/pull/17752
This adds some of the missing flags from https://cloud.google.com/dataflow/docs/reference/pipeline-options to the Go Sdk.
The diff is rough in places because of indentation, but the flags added are:
- dataflow_service_options
- number_of_worker_harness_threads
- dataflow_kms_key
- flexrs_goal
- enable_hot_key_logging
I tried running a pipeline with each of these enabled, though I wasn't able to validate behavior for most because they're not totally visible behavior changes, or they require pretty complex and/or expensive setups to notice a difference.
Notably, this PR doesn't include the following flags which are included on that page:
- worker_disk_type - already exists on the Go Sdk as disk_type (but is currently not documented)
- update - Jack has a PR for this in flight - #17747
- create_from_snapshot - this flag is not useful until we support state
- sdk_location - this is currently a python only option which I don't think we can just get for free. Additional investigation would be needed for this one.
- enable_streaming_engine - I tried this (it should control some experiments), and we don't get it for free with the Go Sdk - we'd have to follow up with the Dataflow team to figure out what it takes to turn streaming engine on for the Go Sdk.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137150512
# [Codecov](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#17752](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cb74f8b) into [master](https://codecov.io/gh/apache/beam/commit/bb8dafb0ecf4643a4c7fa7b28d8fb6ccdc297257?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb8dafb) will **decrease** coverage by `0.00%`.
> The diff coverage is `72.34%`.
```diff
@@ Coverage Diff @@
## master #17752 +/- ##
==========================================
- Coverage 74.00% 73.99% -0.01%
==========================================
Files 695 695
Lines 91798 91820 +22
==========================================
+ Hits 67931 67941 +10
- Misses 22621 22632 +11
- Partials 1246 1247 +1
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `50.43% <72.34%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/17752/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | `22.10% <0.00%> (-0.75%)` | :arrow_down: |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/17752/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | `55.48% <91.89%> (+1.84%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb8dafb...cb74f8b](https://codecov.io/gh/apache/beam/pull/17752?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137195203
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @jrmccluskey for label go.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137147691
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137147690
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137342684
R: @youngoli for final approval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
lostluck commented on code in PR #17752:
URL: https://github.com/apache/beam/pull/17752#discussion_r881850464
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -51,26 +51,31 @@ import (
// TODO(herohde) 5/16/2017: the Dataflow flags should match the other SDKs.
var (
- endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
- stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
- image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
- labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
- serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
- numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
- maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
- diskSizeGb = flag.Int64("disk_size_gb", 0, "Size of root disk for VMs, in GB (optional).")
- diskType = flag.String("disk_type", "", "Type of root disk for VMs (optional).")
- autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
- zone = flag.String("zone", "", "GCP zone (optional)")
- network = flag.String("network", "", "GCP network (optional)")
- subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
- noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
- tempLocation = flag.String("temp_location", "", "Temp location (optional)")
- machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
- minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
- workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
- workerRegion = flag.String("worker_region", "", "Dataflow worker region (optional)")
- workerZone = flag.String("worker_zone", "", "Dataflow worker zone (optional)")
+ endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
+ stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
+ image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
+ labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
+ serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
+ numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
+ workerHarnessThreads = flag.Int64("number_of_worker_harness_threads", 0, "The number of threads per each worker harness process (optional).")
+ maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
+ diskSizeGb = flag.Int64("disk_size_gb", 0, "Size of root disk for VMs, in GB (optional).")
+ diskType = flag.String("disk_type", "", "Type of root disk for VMs (optional).")
+ autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
+ zone = flag.String("zone", "", "GCP zone (optional)")
+ kmsKey = flag.String("dataflow_kms_key", "", "The Cloud KMS key identifier used to encrypt data at rest (optional).")
+ network = flag.String("network", "", "GCP network (optional)")
+ subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
+ noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
+ tempLocation = flag.String("temp_location", "", "Temp location (optional)")
+ machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
+ minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
+ workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
+ workerRegion = flag.String("worker_region", "", "Dataflow worker region (optional)")
+ workerZone = flag.String("worker_zone", "", "Dataflow worker zone (optional)")
+ dataflowServiceOptions = flag.String("dataflow_service_options", "", "Comma separated list of additional job modes and configurations (optional)")
+ flexRSGoal = flag.String("flexrs_goal", "", "Which Flexible Resource Scheduling mode to run in (optional)")
+ enableHotKeyLogging = flag.Bool("enable_hot_key_logging", false, "Specifies that when a hot key is detected in the pipeline, the literal, human-readable key is printed in the user's Cloud Logging project (optional).")
Review Comment:
Filed https://issues.apache.org/jira/browse/BEAM-14512 about below. We should comment this out for now and link to this JIRA. We can keep the rest of the plumbing though, as we'll need it later.
We can't support this yet because we don't support the "ToString" URN/Capability yet. It would be very similar to Drain, but without the user customization. Basically we add an exec Node that takes an element and uses fmt.Sprintf("%v") to turn it into a string. The element comes from a DataSource as usual, and is returned to the service with a DataSink, so it really is just "understand the URN" and "include it in the Capabilities list" (statically, all the time even).
https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L329
Unfortunately, this one is annoying to test because there's no closed form test runner for it. It's not hard to write a job that HotKeys at least, and that we're printing whatever the `String()` method provides (as handled by the `fmt` package).
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -90,29 +95,34 @@ func init() {
// New flags that are already put into pipeline options
// should be added to this map.
var flagFilter = map[string]bool{
- "dataflow_endpoint": true,
- "staging_location": true,
- "worker_harness_container_image": true,
- "labels": true,
- "service_account_email": true,
- "num_workers": true,
- "max_num_workers": true,
- "disk_size_gb": true,
- "disk_type": true,
- "autoscaling_algorithm": true,
- "zone": true,
- "network": true,
- "subnetwork": true,
- "no_use_public_ips": true,
- "temp_location": true,
- "worker_machine_type": true,
- "min_cpu_platform": true,
- "dataflow_worker_jar": true,
- "worker_region": true,
- "worker_zone": true,
- "teardown_policy": true,
- "cpu_profiling": true,
- "session_recording": true,
+ "dataflow_endpoint": true,
+ "staging_location": true,
+ "worker_harness_container_image": true,
+ "labels": true,
+ "service_account_email": true,
+ "num_workers": true,
+ "max_num_workers": true,
+ "number_of_worker_harness_threads": true,
Review Comment:
This list is largely to filter out things that already show up visibly in the UI when set. If it's not UI visible by default, we probably shouldn't filter them out.
My recommendation here is *not* adding anything to start and we can filter them if we notice they stutter in the UI. Ultimately, this helps users (and us) know the configuration of a job, and failing in a "it shows up twice" is fine, but hiding the information could be painful.
I'll also note that if you tried these things already and noticed the stutter, then this comment can be disregarded.
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -242,6 +252,12 @@ func getJobOptions(ctx context.Context) (*dataflowlib.JobOptions, error) {
}
}
+ if *flexRSGoal != "" {
+ if *flexRSGoal != "FLEXRS_UNSPECIFIED" && *flexRSGoal != "FLEXRS_SPEED_OPTIMIZED" && *flexRSGoal != "FLEXRS_COST_OPTIMIZED" {
+ return nil, errors.New("invalid flex resource scheduling goal. Use --flexrs_goal=(FLEXRS_UNSPECIFIED|FLEXRS_SPEED_OPTIMIZED|FLEXRS_COST_OPTIMIZED)")
+ }
Review Comment:
Nit: The switch syntax, while it inverts the case, is a bit clearer since it avoids repeating the `*flexRSGoal !=` several times.
Also, print out the flag value we did get, which makes it easier for users to spot typos.
```suggestion
switch *flexRSGoal {
case "FLEXRS_UNSPECIFIED" , "FLEXRS_SPEED_OPTIMIZED", "FLEXRS_COST_OPTIMIZED":
// valid values
default:
return nil, errors.Errorf("invalid flex resource scheduling goal. Got %q; Use --flexrs_goal=(FLEXRS_UNSPECIFIED|FLEXRS_SPEED_OPTIMIZED|FLEXRS_COST_OPTIMIZED)", *flexRSGoal)
}
```
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -51,26 +51,31 @@ import (
// TODO(herohde) 5/16/2017: the Dataflow flags should match the other SDKs.
var (
- endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
- stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
- image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
- labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
- serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
- numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
- maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
- diskSizeGb = flag.Int64("disk_size_gb", 0, "Size of root disk for VMs, in GB (optional).")
- diskType = flag.String("disk_type", "", "Type of root disk for VMs (optional).")
- autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
- zone = flag.String("zone", "", "GCP zone (optional)")
- network = flag.String("network", "", "GCP network (optional)")
- subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
- noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
- tempLocation = flag.String("temp_location", "", "Temp location (optional)")
- machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
- minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
- workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
- workerRegion = flag.String("worker_region", "", "Dataflow worker region (optional)")
- workerZone = flag.String("worker_zone", "", "Dataflow worker zone (optional)")
+ endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
+ stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
+ image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
Review Comment:
```suggestion
image = flag.String("worker_harness_container_image", "", "Worker harness container image (optional).")
```
Definitely optional at this stage. I haven't used the flag in so long it doesn't show up in my command line history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
lostluck merged PR #17752:
URL: https://github.com/apache/beam/pull/17752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137529541
I didn't know dataflow exposed a threads per worker count. That's *very* exciting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137479686
> * `sdk_location` - this is currently a python only option which I don't think we can just get for free. Additional investigation would be needed for this one.
> * `enable_streaming_engine` - I tried this (it should control some experiments), and we don't get it for free with the Go Sdk - we'd have to follow up with the Dataflow team to figure out what it takes to turn streaming engine on for the Go Sdk.
`sdk_location` is definitely not going to be useful for Go: Everything's already compiled into the binary, so we can't have that overridden at runtime. No action required, as it's unsupportable.
`enable_streaming_engine` - I thought that should already be enabled by default for all Dataflow jobs, but I could be getting that mixed up with Shuffle Service. This can already be controlled by an experiments flag at least. https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#streaming-engine Agreed that additional investigation in the near term is worthwhile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #17752:
URL: https://github.com/apache/beam/pull/17752#discussion_r881905113
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -51,26 +51,31 @@ import (
// TODO(herohde) 5/16/2017: the Dataflow flags should match the other SDKs.
var (
- endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
- stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
- image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
- labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
- serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
- numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
- maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
- diskSizeGb = flag.Int64("disk_size_gb", 0, "Size of root disk for VMs, in GB (optional).")
- diskType = flag.String("disk_type", "", "Type of root disk for VMs (optional).")
- autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
- zone = flag.String("zone", "", "GCP zone (optional)")
- network = flag.String("network", "", "GCP network (optional)")
- subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
- noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
- tempLocation = flag.String("temp_location", "", "Temp location (optional)")
- machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
- minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
- workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
- workerRegion = flag.String("worker_region", "", "Dataflow worker region (optional)")
- workerZone = flag.String("worker_zone", "", "Dataflow worker zone (optional)")
+ endpoint = flag.String("dataflow_endpoint", "", "Dataflow endpoint (optional).")
+ stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
+ image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
+ labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
+ serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
+ numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
+ workerHarnessThreads = flag.Int64("number_of_worker_harness_threads", 0, "The number of threads per each worker harness process (optional).")
+ maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
+ diskSizeGb = flag.Int64("disk_size_gb", 0, "Size of root disk for VMs, in GB (optional).")
+ diskType = flag.String("disk_type", "", "Type of root disk for VMs (optional).")
+ autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
+ zone = flag.String("zone", "", "GCP zone (optional)")
+ kmsKey = flag.String("dataflow_kms_key", "", "The Cloud KMS key identifier used to encrypt data at rest (optional).")
+ network = flag.String("network", "", "GCP network (optional)")
+ subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
+ noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
+ tempLocation = flag.String("temp_location", "", "Temp location (optional)")
+ machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
+ minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
+ workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
+ workerRegion = flag.String("worker_region", "", "Dataflow worker region (optional)")
+ workerZone = flag.String("worker_zone", "", "Dataflow worker zone (optional)")
+ dataflowServiceOptions = flag.String("dataflow_service_options", "", "Comma separated list of additional job modes and configurations (optional)")
+ flexRSGoal = flag.String("flexrs_goal", "", "Which Flexible Resource Scheduling mode to run in (optional)")
+ enableHotKeyLogging = flag.Bool("enable_hot_key_logging", false, "Specifies that when a hot key is detected in the pipeline, the literal, human-readable key is printed in the user's Cloud Logging project (optional).")
Review Comment:
Done!
##########
sdks/go/pkg/beam/runners/dataflow/dataflow.go:
##########
@@ -90,29 +95,34 @@ func init() {
// New flags that are already put into pipeline options
// should be added to this map.
var flagFilter = map[string]bool{
- "dataflow_endpoint": true,
- "staging_location": true,
- "worker_harness_container_image": true,
- "labels": true,
- "service_account_email": true,
- "num_workers": true,
- "max_num_workers": true,
- "disk_size_gb": true,
- "disk_type": true,
- "autoscaling_algorithm": true,
- "zone": true,
- "network": true,
- "subnetwork": true,
- "no_use_public_ips": true,
- "temp_location": true,
- "worker_machine_type": true,
- "min_cpu_platform": true,
- "dataflow_worker_jar": true,
- "worker_region": true,
- "worker_zone": true,
- "teardown_policy": true,
- "cpu_profiling": true,
- "session_recording": true,
+ "dataflow_endpoint": true,
+ "staging_location": true,
+ "worker_harness_container_image": true,
+ "labels": true,
+ "service_account_email": true,
+ "num_workers": true,
+ "max_num_workers": true,
+ "number_of_worker_harness_threads": true,
Review Comment:
Yeah, that's a good call. I removed the new items
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17752: [BEAM-14509] Add several flags to dataflow runner
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17752:
URL: https://github.com/apache/beam/pull/17752#issuecomment-1137147695
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org