You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/17 01:29:09 UTC
[GitHub] [beam] youngoli opened a new pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
youngoli opened a new pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144
Added SDF detection and method validation when creating DoFns. This
adds all the validation except for anything to do with restriction
trackers, which will be added in a later change.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lostluck commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394618692
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -196,13 +217,31 @@ func (f *DoFn) Name() string {
// IsSplittable returns whether the DoFn is a valid Splittable DoFn.
func (f *DoFn) IsSplittable() bool {
- return false // TODO(BEAM-3301): Implement this when we add SDFs.
+ isSdf, _ := validateSdfMethodsPresent((*Fn)(f))
Review comment:
A semantic note: the error shouldn't be ignored. Semantically if a function returns an error, the non error return values are not guaranteed to be valid.
So conventionally, the way to handle IsSplittable here is to check for the error, and return false if not nil, and otherwise return what the boolean is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on issue #11144: [BEAM-3301] Perform SDF
validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on issue #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#issuecomment-599832332
Btw, only major thing I was unsure of. Should I be hiding this functionality behind an experiment flag of some kind? This initial one doesn't do so, and I don't expect people would randomly use it before it's ready because there's no user-facing documentation indicating its usage in the "beam" package, but it's still technically possible to use it right now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783450
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs []funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all subsequent
+ // methods to that result. If there's a mismatch, then we only fail after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
+ var first, fail bool
+
+ for i, name := range sdfNames {
+ _, ok := fn.methods[name]
+ if !ok {
+ missing = append(missing, name)
+ }
+
+ if i == 0 {
+ first = ok
+ } else if ok != first {
+ fail = true
+ }
+ }
+
+ if fail {
+ err := errors.Errorf("not all SplittableDoFn methods are present. Missing methods: %v", missing)
+ return false, err
+ }
+
+ return first, nil
Review comment:
That looks much more readable and easy to follow than the current one. Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli merged pull request #11144: [BEAM-3301] Perform SDF
validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli merged pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on issue #11144: [BEAM-3301] Perform SDF
validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on issue #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#issuecomment-599831761
R: @lostluck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lostluck commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r395339195
##########
File path: sdks/go/pkg/beam/core/graph/fn_test.go
##########
@@ -470,6 +542,169 @@ func (fn *BadDoFnAmbiguousSideInput) StartBundle(bool) {
func (fn *BadDoFnAmbiguousSideInput) FinishBundle(bool) {
}
+// Examples of correct SplittableDoFn signatures
+
+type RestT struct{}
+
+type GoodSdf struct {
Review comment:
Makes sense! Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783430
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs []funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all subsequent
+ // methods to that result. If there's a mismatch, then we only fail after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
Review comment:
Done. I forgot that append works on nil slices.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783417
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -196,13 +217,31 @@ func (f *DoFn) Name() string {
// IsSplittable returns whether the DoFn is a valid Splittable DoFn.
func (f *DoFn) IsSplittable() bool {
- return false // TODO(BEAM-3301): Implement this when we add SDFs.
+ isSdf, _ := validateSdfMethodsPresent((*Fn)(f))
Review comment:
Good point, I'll keep it in mind for the future. For this one, though, I realized it's better just to check for the presence of a single SDF method anyway, rather than calling that validation method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lostluck commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394728377
##########
File path: sdks/go/pkg/beam/core/graph/fn_test.go
##########
@@ -470,6 +542,169 @@ func (fn *BadDoFnAmbiguousSideInput) StartBundle(bool) {
func (fn *BadDoFnAmbiguousSideInput) FinishBundle(bool) {
}
+// Examples of correct SplittableDoFn signatures
+
+type RestT struct{}
+
+type GoodSdf struct {
Review comment:
Shouldn't the ProcessElement method be adjusted here for the Restriction tracker parameter for a "GoodSdf" ?
I guess technically if we don't care about dynamic splitting at all, it should be allowed. It's not unreasonable to prevent it until we have the right idea how to do that, but if so, lets put a TODO in here somewhere to make the intent explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] youngoli commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783477
##########
File path: sdks/go/pkg/beam/core/graph/fn_test.go
##########
@@ -470,6 +542,169 @@ func (fn *BadDoFnAmbiguousSideInput) StartBundle(bool) {
func (fn *BadDoFnAmbiguousSideInput) FinishBundle(bool) {
}
+// Examples of correct SplittableDoFn signatures
+
+type RestT struct{}
+
+type GoodSdf struct {
Review comment:
I was just skipping it because this commit hasn't added restriction trackers yet, but adding a TODO to remind me to change it later is a good idea. Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lostluck commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394725254
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs []funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all subsequent
+ // methods to that result. If there's a mismatch, then we only fail after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
+ var first, fail bool
+
+ for i, name := range sdfNames {
+ _, ok := fn.methods[name]
+ if !ok {
+ missing = append(missing, name)
+ }
+
+ if i == 0 {
+ first = ok
+ } else if ok != first {
+ fail = true
+ }
+ }
+
+ if fail {
+ err := errors.Errorf("not all SplittableDoFn methods are present. Missing methods: %v", missing)
+ return false, err
+ }
+
+ return first, nil
Review comment:
I see that this function is trying to distinguish between partial coverage or complete coverage, but I think it could be simpler.
Consider that the booleans could be removed by comparing whether len(missing) == len(sdfNames) to check if it's simply not an SDF at all (and thus, no error should be returned).
```
switch len(missing) {
case 0:
return true, nil
case len(sdfNames):
return false, nil
default:
err := errors.Errorf("not all SplittableDoFn methods are present. Missing methods: %v", missing)
return false, err
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lostluck commented on a change in pull request #11144:
[BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394722487
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs []funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all subsequent
+ // methods to that result. If there's a mismatch, then we only fail after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
Review comment:
Prefer declaring empty slices using the var syntax.
var missing []string
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services