You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/06 22:27:42 UTC

[GitHub] [beam] youngoli opened a new pull request #11327: [BEAM-9642] Add SDF execution units.

youngoli opened a new pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327
 
 
   Adds the three units needed to execute expanded SDF transforms. Note
   that this is still missing a unit to execute unexpanded SDF transforms,
   and that these units are not currently created in graph translation,
   so this does not actually enable SDFs to be used just yet.
   
   Actual testing of SDFs needs to be done before enabling the graph
   translation changes, but those will be coming next once any bugs
   that might pop up are ironed out.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
   XLang | --- | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) 
   Portable | --- | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407828960
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_test.go
 ##########
 @@ -273,7 +287,9 @@ func (fn *KvSdf) CreateTracker(rest Restriction) *RTracker {
 	return &RTracker{rest, 2}
 }
 
-// ProcessElement is a no-op, it's only included to pass validation.
-func (fn *KvSdf) ProcessElement(*RTracker, int, int) int {
-	return 0
+// ProcessElement emits two ints. The first is the first input (key) +
+// RTracker.Rest.Val. The second is the second input (value) + RTracker.Val.
+func (fn *KvSdf) ProcessElement(rt *RTracker, i1 int, i2 int, emit func(int, int)) {
+	emit(i1+rt.Rest.Val, i2+rt.Val)
+	return
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406402446
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+}
+
+// TestSplitAndSizeRestrictions verifies that the SplitAndSizeRestrictions node
+// both fulfills its contract and successfully invokes the accompanying SDF's
+// SplitRestriction and RestrictionSize methods to do so.
+func TestSplitAndSizeRestrictions(t *testing.T) {
+}
+
+// TestProcessSizedElementsAndRestrictions verifies that the
+// ProcessSizedElementsAndRestrictions node both fulfills its contract and
+// successfully invokes the accompanying SDF's CreateTracker and ProcessElement
+// methods to do so.
+func TestProcessSizedElementsAndRestrictions(t *testing.T) {
+}
+
+// FakeNode is used to capture the outputs of the node being tested by
+// TestSdfNodes. FakeNode appends each element that it receives in its
+// ProcessElement method to its Vals struct, which can then be read by the test.
+type FakeNode struct {
 
 Review comment:
   Why write a new one rather than re-use CaptureNode? https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/unit_test.go#L28

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407827873
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
 ##########
 @@ -0,0 +1,297 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"fmt"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"path"
+
+	"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+)
+
+// PairWithRestriction is an executor for the expanded SDF step of the same
+// name. This is the first step of an expanded SDF. It pairs each main input
+// element with a restriction via the SDF's associated sdf.RestrictionProvider.
+// This step is followed by SplitAndSizeRestrictions.
+type PairWithRestriction struct {
+	UID UnitID
+	Fn  *graph.DoFn
+	Out []Node
+
+	inv *cirInvoker
+}
+
+// ID returns the UnitID for this unit.
+func (n *PairWithRestriction) ID() UnitID {
+	return n.UID
+}
+
+// Up performs one-time setup for this executor.
+func (n *PairWithRestriction) Up(ctx context.Context) error {
+	fn := (*graph.SplittableDoFn)(n.Fn).CreateInitialRestrictionFn()
+	var err error
+	if n.inv, err = newCreateInitialRestrictionInvoker(fn); err != nil {
+		return errors.WithContextf(err, "PairWithRestriction transform with UID %v", n.ID())
+	}
+	return nil
+}
+
+// StartBundle currently does nothing.
+func (n *PairWithRestriction) StartBundle(ctx context.Context, id string, data DataContext) error {
+	return n.Out[0].StartBundle(ctx, id, data)
+}
+
+// ProcessElement expects elm to be the main input to the ParDo. See
+// exec.FullValue for more details on the expected input.
+//
+// ProcessElement creates an initial restriction representing the entire input.
+// The output is in the structure <elem, restriction>, where elem is the main
+// input originally passed in (i.e. the parameter elm). Windows and Timestamp
+// are copied to the outer *FullValue. They still remain within the original
+// element as well, but will no longer be used.
+//
+// Output Diagram:
+//
+//   *FullValue {
+//     Elm: *FullValue (original input)
+//     Elm2: Restriction
+//     Windows
+//     Timestamps
+//   }
+func (n *PairWithRestriction) ProcessElement(ctx context.Context, elm *FullValue, values ...ReStream) error {
+	rest := n.inv.Invoke(elm)
+	output := FullValue{Elm: elm, Elm2: rest, Timestamp: elm.Timestamp, Windows: elm.Windows}
+
+	return n.Out[0].ProcessElement(ctx, &output, values...)
+}
+
+// FinishBundle does some teardown for the end of the bundle.
+func (n *PairWithRestriction) FinishBundle(ctx context.Context) error {
+	n.inv.Reset()
+	return n.Out[0].FinishBundle(ctx)
+}
+
+// Down currently does nothing.
+func (n *PairWithRestriction) Down(ctx context.Context) error {
+	return nil
+}
+
+// String outputs a human-readable description of this transform.
+func (n *PairWithRestriction) String() string {
+	return fmt.Sprintf("SDF.PairWithRestriction[%v] Out:%v", path.Base(n.Fn.Name()), IDs(n.Out...))
+}
+
+// SplitAndSizeRestrictions is an executor for the expanded SDF step of the
+// same name. It is the second step of the expanded SDF, occuring after
+// CreateInitialRestriction. It performs initial splits on the initial restrictions
+// and adds sizing information, producing one or more output elements per input
+// element. This step is followed by ProcessSizedElementsAndRestrictions.
+type SplitAndSizeRestrictions struct {
+	UID UnitID
+	Fn  *graph.DoFn
+	Out []Node
 
 Review comment:
   This was me copying from the old implementation, which had all these nodes wrapping ParDos. No, these shouldn't be outputting to more than one transform, so I'll change this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407843164
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406403109
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+}
+
+// TestSplitAndSizeRestrictions verifies that the SplitAndSizeRestrictions node
+// both fulfills its contract and successfully invokes the accompanying SDF's
+// SplitRestriction and RestrictionSize methods to do so.
+func TestSplitAndSizeRestrictions(t *testing.T) {
+}
+
+// TestProcessSizedElementsAndRestrictions verifies that the
+// ProcessSizedElementsAndRestrictions node both fulfills its contract and
+// successfully invokes the accompanying SDF's CreateTracker and ProcessElement
+// methods to do so.
+func TestProcessSizedElementsAndRestrictions(t *testing.T) {
+}
+
+// FakeNode is used to capture the outputs of the node being tested by
+// TestSdfNodes. FakeNode appends each element that it receives in its
+// ProcessElement method to its Vals struct, which can then be read by the test.
+type FakeNode struct {
+	Vals []*FullValue
 
 Review comment:
   Note, using the pointers in this case has the chance to break these tests if optimizations like FullValue allocation are added to these nodes. Better to copy the FullValue's fields by having plain FVs rather than *FV.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406404105
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
 
 Review comment:
   I forgotwe had cmp probably configured in the gogradle.lock file. Nice. 
   
    https://github.com/apache/beam/blob/master/sdks/go/gogradle.lock#L223
   
   In a different PR we could probably update the comparison helpers in fullvalue_test.go to use cmp options and Transformers instead which would make things much clearer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406400104
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_test.go
 ##########
 @@ -273,7 +287,9 @@ func (fn *KvSdf) CreateTracker(rest Restriction) *RTracker {
 	return &RTracker{rest, 2}
 }
 
-// ProcessElement is a no-op, it's only included to pass validation.
-func (fn *KvSdf) ProcessElement(*RTracker, int, int) int {
-	return 0
+// ProcessElement emits two ints. The first is the first input (key) +
+// RTracker.Rest.Val. The second is the second input (value) + RTracker.Val.
+func (fn *KvSdf) ProcessElement(rt *RTracker, i1 int, i2 int, emit func(int, int)) {
+	emit(i1+rt.Rest.Val, i2+rt.Val)
+	return
 
 Review comment:
   The empty return isn't necessary and can be elided.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on issue #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on issue #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#issuecomment-610069859
 
 
   Run Go PostCommit

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407842781
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+}
+
+// TestSplitAndSizeRestrictions verifies that the SplitAndSizeRestrictions node
+// both fulfills its contract and successfully invokes the accompanying SDF's
+// SplitRestriction and RestrictionSize methods to do so.
+func TestSplitAndSizeRestrictions(t *testing.T) {
+}
+
+// TestProcessSizedElementsAndRestrictions verifies that the
+// ProcessSizedElementsAndRestrictions node both fulfills its contract and
+// successfully invokes the accompanying SDF's CreateTracker and ProcessElement
+// methods to do so.
+func TestProcessSizedElementsAndRestrictions(t *testing.T) {
+}
+
+// FakeNode is used to capture the outputs of the node being tested by
+// TestSdfNodes. FakeNode appends each element that it receives in its
+// ProcessElement method to its Vals struct, which can then be read by the test.
+type FakeNode struct {
 
 Review comment:
   Didn't know that existed (or that whole file, really). Switched to it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck merged pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407829916
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
 
 Review comment:
   Oooh, that seems like a good idea. Made a bug for it: https://jira.apache.org/jira/browse/BEAM-9753

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r405097712
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
 ##########
 @@ -120,11 +120,17 @@ func (n *ParDo) ProcessElement(ctx context.Context, elm *FullValue, values ...Re
 	if n.status != Active {
 		return errors.Errorf("invalid status for pardo %v: %v, want Active", n.UID, n.status)
 	}
+
+	return n.ProcessMainInput(&MainInput{Key: *elm, Values: values})
+}
+
+func (n *ParDo) ProcessMainInput(mainIn *MainInput) error {
 
 Review comment:
   consider not exporting this, as it's helper function for ProcessElement for the SDF components. Note that since they're in the same package, the unexported methods could be used.
   
   Regardless of if it's exported or not, please add a godoc comment why this needs to be separated out.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406398189
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
 ##########
 @@ -0,0 +1,297 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"fmt"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"path"
+
+	"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+)
+
+// PairWithRestriction is an executor for the expanded SDF step of the same
+// name. This is the first step of an expanded SDF. It pairs each main input
+// element with a restriction via the SDF's associated sdf.RestrictionProvider.
+// This step is followed by SplitAndSizeRestrictions.
+type PairWithRestriction struct {
+	UID UnitID
+	Fn  *graph.DoFn
+	Out []Node
+
+	inv *cirInvoker
+}
+
+// ID returns the UnitID for this unit.
+func (n *PairWithRestriction) ID() UnitID {
+	return n.UID
+}
+
+// Up performs one-time setup for this executor.
+func (n *PairWithRestriction) Up(ctx context.Context) error {
+	fn := (*graph.SplittableDoFn)(n.Fn).CreateInitialRestrictionFn()
+	var err error
+	if n.inv, err = newCreateInitialRestrictionInvoker(fn); err != nil {
+		return errors.WithContextf(err, "PairWithRestriction transform with UID %v", n.ID())
+	}
+	return nil
+}
+
+// StartBundle currently does nothing.
+func (n *PairWithRestriction) StartBundle(ctx context.Context, id string, data DataContext) error {
+	return n.Out[0].StartBundle(ctx, id, data)
+}
+
+// ProcessElement expects elm to be the main input to the ParDo. See
+// exec.FullValue for more details on the expected input.
+//
+// ProcessElement creates an initial restriction representing the entire input.
+// The output is in the structure <elem, restriction>, where elem is the main
+// input originally passed in (i.e. the parameter elm). Windows and Timestamp
+// are copied to the outer *FullValue. They still remain within the original
+// element as well, but will no longer be used.
+//
+// Output Diagram:
+//
+//   *FullValue {
+//     Elm: *FullValue (original input)
+//     Elm2: Restriction
+//     Windows
+//     Timestamps
+//   }
+func (n *PairWithRestriction) ProcessElement(ctx context.Context, elm *FullValue, values ...ReStream) error {
+	rest := n.inv.Invoke(elm)
+	output := FullValue{Elm: elm, Elm2: rest, Timestamp: elm.Timestamp, Windows: elm.Windows}
+
+	return n.Out[0].ProcessElement(ctx, &output, values...)
+}
+
+// FinishBundle does some teardown for the end of the bundle.
+func (n *PairWithRestriction) FinishBundle(ctx context.Context) error {
+	n.inv.Reset()
+	return n.Out[0].FinishBundle(ctx)
+}
+
+// Down currently does nothing.
+func (n *PairWithRestriction) Down(ctx context.Context) error {
+	return nil
+}
+
+// String outputs a human-readable description of this transform.
+func (n *PairWithRestriction) String() string {
+	return fmt.Sprintf("SDF.PairWithRestriction[%v] Out:%v", path.Base(n.Fn.Name()), IDs(n.Out...))
+}
+
+// SplitAndSizeRestrictions is an executor for the expanded SDF step of the
+// same name. It is the second step of the expanded SDF, occuring after
+// CreateInitialRestriction. It performs initial splits on the initial restrictions
+// and adds sizing information, producing one or more output elements per input
+// element. This step is followed by ProcessSizedElementsAndRestrictions.
+type SplitAndSizeRestrictions struct {
+	UID UnitID
+	Fn  *graph.DoFn
+	Out []Node
 
 Review comment:
   Is it possible for SplitAndSizeRestrictions or PairWithRestriction to output to more than one other transform? Why have a slice?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407842880
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+}
+
+// TestSplitAndSizeRestrictions verifies that the SplitAndSizeRestrictions node
+// both fulfills its contract and successfully invokes the accompanying SDF's
+// SplitRestriction and RestrictionSize methods to do so.
+func TestSplitAndSizeRestrictions(t *testing.T) {
+}
+
+// TestProcessSizedElementsAndRestrictions verifies that the
+// ProcessSizedElementsAndRestrictions node both fulfills its contract and
+// successfully invokes the accompanying SDF's CreateTracker and ProcessElement
+// methods to do so.
+func TestProcessSizedElementsAndRestrictions(t *testing.T) {
+}
+
+// FakeNode is used to capture the outputs of the node being tested by
+// TestSdfNodes. FakeNode appends each element that it receives in its
+// ProcessElement method to its Vals struct, which can then be read by the test.
+type FakeNode struct {
+	Vals []*FullValue
 
 Review comment:
   Noted for the future.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r406407836
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
 ##########
 @@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+	"context"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+	"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+	"github.com/google/go-cmp/cmp"
+	"testing"
+)
+
+// testTimestamp is a constant used to check that timestamps are retained.
+const testTimestamp = 15
+
+// testWindow is a constant used to check that windows are retained
+var testWindows = []typex.Window{window.IntervalWindow{Start: 10, End: 20}}
+
+// TestSdfNodes verifies that the various SDF nodes fulfill each of their
+// described contracts, that they each successfully invoke any SDF methods
+// needed, and that they preserve timestamps and windows correctly.
+func TestSdfNodes(t *testing.T) {
+	// Setup. The DoFns created below are defined in sdf_invokers_test.go and
+	// have testable behavior to confirm that they got correctly invoked.
+	// Without knowing the expected behavior of these DoFns, the desired outputs
+	// in the unit tests below will not make much sense.
+	dfn, err := graph.NewDoFn(&Sdf{}, graph.NumMainInputs(graph.MainSingle))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+	kvdfn, err := graph.NewDoFn(&KvSdf{}, graph.NumMainInputs(graph.MainKv))
+	if err != nil {
+		t.Fatalf("invalid function: %v", err)
+	}
+
+	// Validate PairWithRestriction matches its contract and properly invokes
+	// SDF method CreateInitialRestriction.
+	t.Run("PairWithRestriction", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      nil,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm:       5,
+					Elm2:      2,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm: &FullValue{
+						Elm:       5,
+						Elm2:      2,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{7},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := PairWithRestriction{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				got := fake.Vals[0]
+				if !cmp.Equal(got, test.want) {
+					t.Errorf("ProcessElement(%v) has incorrect output: got: %v, want: %v",
+						test.in, got, test.want)
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate SplitAndSizeRestrictions matches its contract and properly
+	// invokes SDF methods SplitRestriction and RestrictionSize.
+	t.Run("SplitAndSizeRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want []*FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      nil,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{5},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{7},
+						},
+						Elm2:      9.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      nil,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      10.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm:       2,
+						Elm2:      5,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					Elm2:      Restriction{3},
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: []*FullValue{
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{5},
+						},
+						Elm2:      12.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+					{
+						Elm: &FullValue{
+							Elm: &FullValue{
+								Elm:       2,
+								Elm2:      5,
+								Timestamp: testTimestamp,
+								Windows:   testWindows,
+							},
+							Elm2: Restriction{8},
+						},
+						Elm2:      15.0,
+						Timestamp: testTimestamp,
+						Windows:   testWindows,
+					},
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				node := SplitAndSizeRestrictions{UID: 0, Fn: test.fn, Out: []Node{fake}}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
+					t.Fatalf("ProcessElement failed: %v", err)
+				}
+				for i, got := range fake.Vals {
+					if !cmp.Equal(got, test.want[i]) {
+						t.Errorf("ProcessElement(%v) has incorrect output %v: got: %v, want: %v",
+							test.in, i, got, test.want)
+					}
+				}
+				if err := node.FinishBundle(ctx); err != nil {
+					t.Fatalf("FinishBundle failed: %v", err)
+				}
+				if err := node.Down(ctx); err != nil {
+					t.Fatalf("Down failed: %v", err)
+				}
+			})
+		}
+	})
+
+	// Validate ProcessSizedElementsAndRestrictions matches its contract and
+	// properly invokes SDF methods CreateTracker and ProcessElement.
+	t.Run("ProcessSizedElementsAndRestrictions", func(t *testing.T) {
+		tests := []struct {
+			name string
+			fn   *graph.DoFn
+			in   *FullValue
+			want *FullValue
+		}{
+			{
+				name: "SingleElem",
+				fn:   dfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      nil,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      8.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      4,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+			{
+				name: "KvElem",
+				fn:   kvdfn,
+				in: &FullValue{
+					Elm: &FullValue{
+						Elm: &FullValue{
+							Elm:       3,
+							Elm2:      10,
+							Timestamp: testTimestamp,
+							Windows:   testWindows,
+						},
+						Elm2: Restriction{5},
+					},
+					Elm2:      18.0,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+				want: &FullValue{
+					Elm:       8,
+					Elm2:      12,
+					Timestamp: testTimestamp,
+					Windows:   testWindows,
+				},
+			},
+		}
+		for _, test := range tests {
+			test := test
+			t.Run(test.name, func(t *testing.T) {
+				ctx := context.Background()
+				fake := &FakeNode{}
+				n := &ParDo{UID: 0, Fn: test.fn, Out: []Node{fake}}
+				node := ProcessSizedElementsAndRestrictions{PDo: n}
+
+				if err := node.Up(ctx); err != nil {
+					t.Fatalf("Up failed: %v", err)
+				}
+				if err := node.StartBundle(ctx, "bundle_id", DataContext{}); err != nil {
+					t.Fatalf("StartBundle failed: %v", err)
+				}
+
+				if err := node.ProcessElement(ctx, test.in); err != nil {
 
 Review comment:
   There's a lot of duplicated code for these tests. Consider using FixedRoot, and the constructAndExecutePlan helper to reduce the duplication. 
   We may want to move the constructAndExecutePlan helper to unit_test.go (in another PR).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#discussion_r407827372
 
 

 ##########
 File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go
 ##########
 @@ -120,11 +120,17 @@ func (n *ParDo) ProcessElement(ctx context.Context, elm *FullValue, values ...Re
 	if n.status != Active {
 		return errors.Errorf("invalid status for pardo %v: %v, want Active", n.UID, n.status)
 	}
+
+	return n.ProcessMainInput(&MainInput{Key: *elm, Values: values})
+}
+
+func (n *ParDo) ProcessMainInput(mainIn *MainInput) error {
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] youngoli commented on issue #11327: [BEAM-9642] Add SDF execution units.

Posted by GitBox <gi...@apache.org>.
youngoli commented on issue #11327: [BEAM-9642] Add SDF execution units.
URL: https://github.com/apache/beam/pull/11327#issuecomment-610070053
 
 
   R: @lostluck 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services