You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/21 01:52:54 UTC
[GitHub] [beam] youngoli opened a new pull request #11474: [BEAM-9642] Enabling SDF exec runners and fixing small bug.
youngoli opened a new pull request #11474:
URL: https://github.com/apache/beam/pull/11474
This actually enables SDFs to actually get used. Only basic testing has
been done, though. One of the bugs that was revealed from that testing
was fixed in the ProcessSizedElementsAndRestrictions step.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on issue #11474: [BEAM-9642] Enabling SDF exec runners and fixing small bug.
Posted by GitBox <gi...@apache.org>.
youngoli commented on issue #11474:
URL: https://github.com/apache/beam/pull/11474#issuecomment-616902881
R: @lostluck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11474: [BEAM-9642] Enabling SDF exec runners and fixing small bug.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11474:
URL: https://github.com/apache/beam/pull/11474#discussion_r412316843
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -263,20 +266,33 @@ func (n *ProcessSizedElementsAndRestrictions) ProcessElement(ctx context.Context
return errors.Errorf("invalid status for ParDo %v: %v, want Active", n.PDo.UID, n.PDo.status)
}
- userElm := elm.Elm.(*FullValue).Elm.(*FullValue)
rest := elm.Elm.(*FullValue).Elm2
rt := n.inv.Invoke(rest)
+ var mainIn = &MainInput{
Review comment:
The idiomatic form for this would be to do the following:
```suggestion
mainIn := &MainInput{
```
var declarations are usually for uninitialized variables.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -263,20 +266,33 @@ func (n *ProcessSizedElementsAndRestrictions) ProcessElement(ctx context.Context
return errors.Errorf("invalid status for ParDo %v: %v, want Active", n.PDo.UID, n.PDo.status)
}
- userElm := elm.Elm.(*FullValue).Elm.(*FullValue)
rest := elm.Elm.(*FullValue).Elm2
rt := n.inv.Invoke(rest)
+ var mainIn = &MainInput{
+ Values: values,
+ RTracker: rt,
+ }
- return n.PDo.processMainInput(&MainInput{
- Key: FullValue{ // User userElm's values but the top-level windows and timestamp.
+ // For the key, the way we fill it out depends on whether the input element
+ // is a KV or single-element. Single-elements might have been lifted out of
+ // their FullValue if they were decoded, so we need to have a case for that.
+ // Also, we use the the top-level windows and timestamp.
+ if userElm, ok := elm.Elm.(*FullValue).Elm.(*FullValue); ok {
Review comment:
Note for the performance pass: We can know KV vs not KV from the input PCollection coderat exec/translate.go time, so we can avoid doing this if branch per element.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
##########
@@ -372,32 +386,43 @@ func (b *builder) makeLink(from string, id linkID) (Node, error) {
switch op {
case graph.ParDo:
- n := &ParDo{UID: b.idgen.New(), Inbound: in, Out: out}
- n.Fn, err = graph.AsDoFn(fn, graph.MainUnknown)
+ dofn, err := graph.AsDoFn(fn, graph.MainUnknown)
if err != nil {
return nil, err
}
- n.PID = transform.GetUniqueName()
-
- input := unmarshalKeyedValues(transform.GetInputs())
- for i := 1; i < len(input); i++ {
- // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
- // For now, assume we don't need any information in the pardo payload.
-
- ec, wc, err := b.makeCoderForPCollection(input[i])
- if err != nil {
- return nil, err
+ switch urn {
+ case urnPairWithRestriction:
+ u = &PairWithRestriction{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ case urnSplitAndSizeRestrictions:
+ u = &SplitAndSizeRestrictions{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ default:
+ n := &ParDo{UID: b.idgen.New(), Fn: dofn, Inbound: in, Out: out}
+ n.PID = transform.GetUniqueName()
+
+ input := unmarshalKeyedValues(transform.GetInputs())
+ for i := 1; i < len(input); i++ {
+ // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
Review comment:
NoActionRequiredNit: I don't like that this gets to 6 indentations deep, which is a bit of a readability concern, but I'm not certain that moving this to a helper function would be that helpful. We'd probably need to break it into two helpers, one from handling the graphx.URNDoFn, and then this graph.ParDo block.
These are a changes for another PR though.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
##########
@@ -372,32 +386,43 @@ func (b *builder) makeLink(from string, id linkID) (Node, error) {
switch op {
case graph.ParDo:
- n := &ParDo{UID: b.idgen.New(), Inbound: in, Out: out}
- n.Fn, err = graph.AsDoFn(fn, graph.MainUnknown)
+ dofn, err := graph.AsDoFn(fn, graph.MainUnknown)
if err != nil {
return nil, err
}
- n.PID = transform.GetUniqueName()
-
- input := unmarshalKeyedValues(transform.GetInputs())
- for i := 1; i < len(input); i++ {
- // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
- // For now, assume we don't need any information in the pardo payload.
-
- ec, wc, err := b.makeCoderForPCollection(input[i])
- if err != nil {
- return nil, err
+ switch urn {
+ case urnPairWithRestriction:
+ u = &PairWithRestriction{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ case urnSplitAndSizeRestrictions:
+ u = &SplitAndSizeRestrictions{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ default:
+ n := &ParDo{UID: b.idgen.New(), Fn: dofn, Inbound: in, Out: out}
+ n.PID = transform.GetUniqueName()
+
+ input := unmarshalKeyedValues(transform.GetInputs())
+ for i := 1; i < len(input); i++ {
+ // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
+ // For now, assume we don't need any information in the pardo payload.
+
+ ec, wc, err := b.makeCoderForPCollection(input[i])
+ if err != nil {
+ return nil, err
+ }
+
+ sid := StreamID{
+ Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
+ PtransformID: id.to,
+ }
+ sideInputID := fmt.Sprintf("i%v", i) // SideInputID (= local id, "iN")
+ side := NewSideInputAdapter(sid, sideInputID, coder.NewW(ec, wc))
+ n.Side = append(n.Side, side)
}
-
- sid := StreamID{
- Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
- PtransformID: id.to,
+ if urn == urnProcessSizedElementsAndRestrictions {
+ u = &ProcessSizedElementsAndRestrictions{PDo: n}
+ } else {
+ u = n
}
Review comment:
Consider moving the common case to always be used, and then corrected afterwards.
```suggestion
u = n
if urn == urnProcessSizedElementsAndRestrictions {
u = &ProcessSizedElementsAndRestrictions{PDo: n}
}
```
This idiom avoids the else block, and indicates that if there were a ?: operator in Go, that's what would happen. The compiler will optimize this appropriately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a change in pull request #11474: [BEAM-9642] Enabling SDF exec runners and fixing small bug.
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11474:
URL: https://github.com/apache/beam/pull/11474#discussion_r412590798
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -263,20 +266,33 @@ func (n *ProcessSizedElementsAndRestrictions) ProcessElement(ctx context.Context
return errors.Errorf("invalid status for ParDo %v: %v, want Active", n.PDo.UID, n.PDo.status)
}
- userElm := elm.Elm.(*FullValue).Elm.(*FullValue)
rest := elm.Elm.(*FullValue).Elm2
rt := n.inv.Invoke(rest)
+ var mainIn = &MainInput{
Review comment:
Whoops, that was left behind in editing. Fixed
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -263,20 +266,33 @@ func (n *ProcessSizedElementsAndRestrictions) ProcessElement(ctx context.Context
return errors.Errorf("invalid status for ParDo %v: %v, want Active", n.PDo.UID, n.PDo.status)
}
- userElm := elm.Elm.(*FullValue).Elm.(*FullValue)
rest := elm.Elm.(*FullValue).Elm2
rt := n.inv.Invoke(rest)
+ var mainIn = &MainInput{
+ Values: values,
+ RTracker: rt,
+ }
- return n.PDo.processMainInput(&MainInput{
- Key: FullValue{ // User userElm's values but the top-level windows and timestamp.
+ // For the key, the way we fill it out depends on whether the input element
+ // is a KV or single-element. Single-elements might have been lifted out of
+ // their FullValue if they were decoded, so we need to have a case for that.
+ // Also, we use the the top-level windows and timestamp.
+ if userElm, ok := elm.Elm.(*FullValue).Elm.(*FullValue); ok {
Review comment:
Ack. Added a TODO to remind me.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
##########
@@ -372,32 +386,43 @@ func (b *builder) makeLink(from string, id linkID) (Node, error) {
switch op {
case graph.ParDo:
- n := &ParDo{UID: b.idgen.New(), Inbound: in, Out: out}
- n.Fn, err = graph.AsDoFn(fn, graph.MainUnknown)
+ dofn, err := graph.AsDoFn(fn, graph.MainUnknown)
if err != nil {
return nil, err
}
- n.PID = transform.GetUniqueName()
-
- input := unmarshalKeyedValues(transform.GetInputs())
- for i := 1; i < len(input); i++ {
- // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
- // For now, assume we don't need any information in the pardo payload.
-
- ec, wc, err := b.makeCoderForPCollection(input[i])
- if err != nil {
- return nil, err
+ switch urn {
+ case urnPairWithRestriction:
+ u = &PairWithRestriction{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ case urnSplitAndSizeRestrictions:
+ u = &SplitAndSizeRestrictions{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ default:
+ n := &ParDo{UID: b.idgen.New(), Fn: dofn, Inbound: in, Out: out}
+ n.PID = transform.GetUniqueName()
+
+ input := unmarshalKeyedValues(transform.GetInputs())
+ for i := 1; i < len(input); i++ {
+ // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
Review comment:
Agreed. I was iffy on it myself since even 5 indentations deep was already getting difficult to follow.
##########
File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
##########
@@ -372,32 +386,43 @@ func (b *builder) makeLink(from string, id linkID) (Node, error) {
switch op {
case graph.ParDo:
- n := &ParDo{UID: b.idgen.New(), Inbound: in, Out: out}
- n.Fn, err = graph.AsDoFn(fn, graph.MainUnknown)
+ dofn, err := graph.AsDoFn(fn, graph.MainUnknown)
if err != nil {
return nil, err
}
- n.PID = transform.GetUniqueName()
-
- input := unmarshalKeyedValues(transform.GetInputs())
- for i := 1; i < len(input); i++ {
- // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
- // For now, assume we don't need any information in the pardo payload.
-
- ec, wc, err := b.makeCoderForPCollection(input[i])
- if err != nil {
- return nil, err
+ switch urn {
+ case urnPairWithRestriction:
+ u = &PairWithRestriction{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ case urnSplitAndSizeRestrictions:
+ u = &SplitAndSizeRestrictions{UID: b.idgen.New(), Fn: dofn, Out: out[0]}
+ default:
+ n := &ParDo{UID: b.idgen.New(), Fn: dofn, Inbound: in, Out: out}
+ n.PID = transform.GetUniqueName()
+
+ input := unmarshalKeyedValues(transform.GetInputs())
+ for i := 1; i < len(input); i++ {
+ // TODO(herohde) 8/8/2018: handle different windows, view_fn and window_mapping_fn.
+ // For now, assume we don't need any information in the pardo payload.
+
+ ec, wc, err := b.makeCoderForPCollection(input[i])
+ if err != nil {
+ return nil, err
+ }
+
+ sid := StreamID{
+ Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
+ PtransformID: id.to,
+ }
+ sideInputID := fmt.Sprintf("i%v", i) // SideInputID (= local id, "iN")
+ side := NewSideInputAdapter(sid, sideInputID, coder.NewW(ec, wc))
+ n.Side = append(n.Side, side)
}
-
- sid := StreamID{
- Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
- PtransformID: id.to,
+ if urn == urnProcessSizedElementsAndRestrictions {
+ u = &ProcessSizedElementsAndRestrictions{PDo: n}
+ } else {
+ u = n
}
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org