You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "hnnsgstfssn (via GitHub)" <gi...@apache.org> on 2023/03/11 10:38:55 UTC
[GitHub] [beam] hnnsgstfssn opened a new pull request, #25808: Add periodic.Sequence and periodic.Impulse transforms
hnnsgstfssn opened a new pull request, #25808:
URL: https://github.com/apache/beam/pull/25808
The new transforms extends support for the slowly updating side input pattern [1] as tracked in [2].
An attempt to mirror the logic of the Python implementation [3] has been made with minor idiomatic changes.
Java [4][5] and Python [6] have influenced the documentation and naming.
[1] https://beam.apache.org/documentation/patterns/side-inputs/
[2] https://github.com/apache/beam/issues/23106
[3] https://github.com/apache/beam/blob/v2.46.0/sdks/python/apache_beam/transforms/periodicsequence.py#L59
[4] https://beam.apache.org/releases/javadoc/2.46.0/org/apache/beam/sdk/transforms/PeriodicSequence.html
[5] https://beam.apache.org/releases/javadoc/2.46.0/org/apache/beam/sdk/transforms/PeriodicImpulse.html
[6] https://beam.apache.org/releases/pydoc/2.46.0/apache_beam.transforms.periodicsequence.html?highlight=periodicimpulse#module-apache_beam.transforms.periodicsequence
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1141423750
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
Review Comment:
It's important to have a blank line after the license, and before the package documentation so the license header isn't the documentation.
I've added a package comment, which should conform to the style guide: https://google.github.io/styleguide/go/decisions#package-comments
```suggestion
// limitations under the License.
// slowly_updating_side_input is an example pipeline demonstrating the pattern described
// at https://beam.apache.org/documentation/patterns/side-inputs/.
package main
```
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
Review Comment:
A little non-standard, but I appreciate how this cleans up and ensures consistency in the default times values.
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
+
+ startTime, _ := time.Parse(time.RFC3339, periodicSequenceStart)
+ endTime, _ := time.Parse(time.RFC3339, periodicSequenceEnd)
+ sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
+ beam.ParDo(
+ s,
+ update,
+ periodic.Impulse(
+ s,
+ startTime,
+ endTime,
+ periodicSequenceInterval,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
The nesting here, has the added deficit of hiding the core of the example.
```suggestion
// Generate an impulse every period.
periodicImp := periodic.Impulse(s, startTime, endTime, periodicSequenceInterval)
// Use the impulse to trigger some other ordinary transform.
updatedImp := beam.ParDo(s, update, periodicImp)
// Window for use as a side input, to allow the input to change with windows.
sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
updatedImp,
beam.Trigger(trigger.Repeat(trigger.Always())),
beam.PanesDiscard(),
)
```
I'll note that the window for the side input is usually going to be larger than the window for the main processing. While this isn't wrong, the usualy goal around the pattern is a situation like allowing files that change hourly get read in once each hour, and have the more frequent data able to re-use the cached read in file. (Granted, this behavior isn't yet enabled by default in the Go SDK, but that's an aside).
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
Review Comment:
This will work in the direct runner, but that's because the direct runner won't successfully run the example or anything local. It would be better to fold things into the sequence definition to enable appropriate testing behavior.
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
Don't do this. This is far less readable than having single use variables, which makes the structure of the pipeline explicit, and linear with reading the program.
Nesting like this leads to needing to read the code to the middle, then back track back up, in order to understand the pipeline, which isn't useful for example code, which will be read way more than written.
```suggestion
source := pubsubio.Read(s, project, inputTopic, nil)
keyedSource := beam.AddFixedKey(s, source) // simulate keyed data by adding a fixed key
mainInput := beam.WindowInto(
s,
keyedSource,
window.NewFixedWindows(periodicSequenceInterval),
beam.Trigger(trigger.Repeat(trigger.Always())),
beam.PanesDiscard(),
)
```
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
Review Comment:
It's a small but critical thing that we clarify this is in event time, since it might not match processing time, depending on how the source is configured.
```suggestion
log.Infof(ctx, "Making external call %d at event time %s", i, t.ToTime().Format(time.RFC3339))
```
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
Review Comment:
We should be able to add a short unit test if using the `prism` runner directly.
While the runner isn't fully complete yet, it *does* run and execute ProcessContinuation transforms and watermarks!
It just doesn't do the splitting just yet, or actually "wait" for any process continuations at the moment. But when the "sequence" is done, it will terminate, so we can add a test with period of a second, a duration of a minute, and then count that we're getting 60 elements out of the transform. (Small risk of getting 59 instead, as a flake...)
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
Review Comment:
Just to explain what happens here and how it works:
The transform will continue until the restriction is fully processed, or the runner has told the bundle to split. On that split, tryClaim returns false, GetError nil, isDone false, and then we reschedule for later with the ResumeProcessingIn continuation. Note that this isn't a hard timer, and a runner may be early or late (usually early).
The output watermark is held to the last output timestamp. When Stop happens, the watermark can then advance to whereever the source is at (probably end of Global Window).
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
Review Comment:
Per previous mention, this can't be serialized so it's non portable (and it's complicated to validate in once it's passed through serialization if it's hacked around). I'd remove it.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
Review Comment:
Since this is just immeadiately aliased in the same package, the code may as well directly export the type. The indirection doesn't add anything but possible confusion to readers in switching between the two ways of referring to the same type.
Per the style guide: Don't use aliases if they aren't needed https://google.github.io/styleguide/go/decisions#type-aliases
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
Review Comment:
I don't expect additional options will be added, so lets match Java and Python with this and use a boolean.
Effectively this transform just produces a PCollection, which means any expansions can be ordinary DoFns or other composites to add other functionality. No need to overcomplicate things here.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
And mTime.Times are prevented from being serialized.
I'd use int64s makes it clear, and it's an implementation detail. (and did so for a similar, but test specific transform, for testing prism). Reduces likely hood of architecture specific issues too since int's width is arch dependent.
TBH it's odd to me that the "value" for the periodic impulses are time instants, which are the event time anyway. I'd prefer we avoid the needless duplication Java and Python take by just having int for Sequence, and []byte{} for Impulse. Int is specific and clear enough for sequence (it's a sequence of numbers, done), but int64 is likely better for clarity on range across architectures. []byte is what the normal Impulse returns, and so it would make it more compatible / drop in with existing transforms that consume the []byte, even if it's ignored.
We can also just encode the time/timestamp into the []byte, if we so choose, but again, it's redundant.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
Review Comment:
It should look at the existing restriction and modify start and end appropriately, but this might be OK for periodic sequence (especially if it's been validated to work as expected on a runner like Google Cloud Dataflow).
Basically the restriction will be immeadiately provided to the ProcessElement method, and that method needs to handle the given restriction correctly, so it's probably nice to know what the last "state" of the restriction is, even if a ProcessContinuation Stop is immeadiately getting returned next.
It's a little un-intuitive, but to avoid dataloss errors, drain can be thought of as simply advancing all root watermarks immeadiately to EndOfGlobalWindow, and allowing the pipeline to terminate processing. The TruncateRestrictions are to re-process current restrictions, and shrink them. Eg. TruncateRestriction is where an IO could comunicate and write to an external system about what hasn't been processed yet, so the next caller of that system can pick up where it was left off. Or a system like Pubsub would effectively handles that case by simply not having any of the outstanding messages acked, so they'll be delivered to the next time the subscriber reads the topic. Different systems, both effective.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
For handling End, we can also just have a variable set to "largest practical time", and use that. It's going to be a very rare case where someone wants to process a stream of data that extends past the year 9999. (Which we could work around by changing the serialization... but that's not urgent now.)
We can also pick something like another arbitrary time with a nanosecond scale difference as a sentinel value that we check for (eg. Zero + 1 nanosecond), which we check for and treat as End of Time. Beam doesn't promise handling event time granularities under microseconds IIRC (if smaller than milliseconds at all...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142345012
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
+
+ startTime, _ := time.Parse(time.RFC3339, periodicSequenceStart)
+ endTime, _ := time.Parse(time.RFC3339, periodicSequenceEnd)
+ sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
+ beam.ParDo(
+ s,
+ update,
+ periodic.Impulse(
+ s,
+ startTime,
+ endTime,
+ periodicSequenceInterval,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
Wasn't able to apply this suggestion after having changed the `periodic.Impulse` signature, but have applied the same change in a separate commit. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1477023778
OK, I think if this is patched in, it should resolve the drain issue. (Specifically the sdf.go file, the test doesn't need changing).
https://github.com/apache/beam/pull/25908
If that works, please review/approve/lgtm that chagne and I'll merge it in, and we can unblock Periodic impulse!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464893546
The draining failure does not happen on latest stable Beam 2.45.0, but does happen on later versions. As of writing 2.46.0 and 2.47.0.dev had the example pipeline fail to drain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464922302
Considering it runs on latest stable as of writing, I'll mark this as ready for review.
I'm still expecting to potentially add tests, if there is a good way to test these streaming features, but it may be acceptable to have manually verified the runs example successfully on Dataflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1144140915
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ register.Function2x0(sequenceToImpulse)
+ register.Emitter1[[]byte]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start int64
+ End int64
Review Comment:
I've added comments/docstrings to those attributes on the `SequenceDefinition` along with a constructor, `NewSequenceDefinnition`, to create it from `time.Time` and `time.Duration` values. Let me know if you would like to alter anything about it before merge.
Thanks Robert!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1477092424
Other example is merged in. Let me know when the test changes are made and I'll do one last pass. If it's good then, I'll merge this in!
Thanks again for your contribution and patience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1133067153
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
Review Comment:
I was wondering if returning a zero value restriction like this does actually correctly truncate the restriction, or if it should instead look at the existing restriction and modify start and end appropriately. If so, should it move start up to or past end, or should move end down to or past start?
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
Review Comment:
While playing around locally I've added this way to mock the time.Now function. If it ends up being unused I'm happy to drop it if that makes more sense.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
Not sure these validations are right. [Python sets](https://github.com/apache/beam/blob/30b26171b4038c1c976adbc3be386e118b35153b/sdks/python/apache_beam/transforms/periodicsequence.py#L160) defaults for both start and end and a seemingly arbitrary interval.
Presumably the max interval is `end.Sub(start)` and end can be clamped to `mtime.MaxTimestamp`, although currently that would not work since the times are serialized using `time.Time` which [will fail](https://pkg.go.dev/time#Time.MarshalJSON) for such high values. This can be avoided by serializing integers.
Please advice some thoughts on what validation is appropriate to add here; I'm happy to make any changes.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
Review Comment:
Initially I added a simpler required `applyWindow bool` argument to `Impulse`, and that might be preferred to this slightly more involved way with functional options. If it is the case that more options are not going to be added for the foreseeable future, that simpler version might be preferred. I'm happy to go either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1475327556
This is failing our Staticheck action. Not hard to fix at least.
https://github.com/apache/beam/actions/runs/4392151772/jobs/7693061461?pr=25808
```
Error: transforms/periodic/periodic.go:61:18: calling math.Ceil on a converted integer is pointless (SA4015)
Error: transforms/periodic/periodic.go:132:6: func withNowFunc is unused (U1000)
Error: Process completed with exit code 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142774721
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
+
+ startTime, _ := time.Parse(time.RFC3339, periodicSequenceStart)
+ endTime, _ := time.Parse(time.RFC3339, periodicSequenceEnd)
+ sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
+ beam.ParDo(
+ s,
+ update,
+ periodic.Impulse(
+ s,
+ startTime,
+ endTime,
+ periodicSequenceInterval,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
I didn't address the larger side input window. Since it's an example, we don't need to over configure things. I'd make the window 5 times larger to demonstrate the paradigm, and an explicit comment that's what the larger window is for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464922884
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464941382
> For the drain issue, Python and Java implementation also did not drain until recent fix (#23765 #25716). Basically the transform should (i) update watermark when advance and (ii) truncate immediately when truncate is called (happens in drain)
Ah! I didn't realize those fixes [1][2] for Python were recent- thanks for highlight it. I tried including both (i) and (ii) here i.e. implementing `TruncateRestriction` and `CreateWatermarkEstimator` and updating the watermark in `ProcessElement`, but might not be doing it correctly.
[1] https://github.com/apache/beam/commit/30b26171b4038c1c976adbc3be386e118b35153b
[2] https://github.com/apache/beam/commit/7dba78d611089a1826c5655ed7ffcdf3f6007eae
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464893889
Note that the example was added mainly for illustration of the code used to manually test this PR on Dataflow. Let me know if it is useful to keep it, or if I should drop it from the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142669478
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
Review Comment:
Oh, but we can't do nil here. I think returning a zero value restriction is correct.
We don't want to return the exact same restruction since we want this to terminate, and if the restriction isn't shortened somehow, it will never drain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1143737540
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ register.Function2x0(sequenceToImpulse)
+ register.Emitter1[[]byte]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start int64
+ End int64
Review Comment:
Please add comments that these represent times in milliseconds.
Also would be good to add a construction function to do this conversion for users from `time.Time` if they're using `periodic.Sequence` themselves rather than through `periodic.Impulse`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #25808:
URL: https://github.com/apache/beam/pull/25808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1476557086
The example is still not able to successfully drain, failing with the following message. I'm unsure why, since as far as I can tell it is both (i) updating the watermark when advancing and (ii) truncating immediately.
<details>
```
"Error message from worker: generic::unknown: process bundle failed for instruction process_bundle-13-7 using plan drain-S02-11 : panic: runtime error: index out of range [2] with length 2
Full error:
while executing Process for Plan[drain-S02-11]:
2: DataSink[S[ptransform-9@localhost:12371]] Coder:W;coder-50<KV;coder-51<int[varintz;c2];coder-52,string;coder-53>>!IWC
3: PCollection[pcollection-32] Out:[2]
4: WindowInto[FIX[1m0s]]. Out:2
5: PCollection[pcollection-26] Out:[4]
6: ParDo[main.update] Out:[5] Sig: func(context.Context, mtime.Time, []uint8, func(int, string))
7: PCollection[pcollection-22] Out:[6]
8: SDF.ProcessSizedElementsAndRestrictions[periodic.sequenceGenDoFn] UID:8 Out:[7]
9: PCollection[pcollection-12-truncate-output] Out:[8]
10: SDF.TruncateSizedRestriction[periodic.sequenceGenDoFn] UID:10 Out:[9]
1: DataSource[S[ptransform-8@localhost:12371], 0] Out:10 Coder:W;coder-38<KV;coder-39<KV;coder-40<LP;coder-41<R[periodic.SequenceDefinition]>,KV;coder-42<offsetrange.Restriction[offsetrange.Restriction;c9];coder-43,bool;coder-44>>,double;coder-45>>!GWC
caused by:
panic: runtime error: index out of range [2] with length 2 goroutine 70 [running]:
runtime/debug.Stack()
/usr/lib/go/src/runtime/debug/stack.go:24 +0x65
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic.func1()
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:58 +0xa5
panic({0xfb1ae0, 0xc000137290})
/usr/lib/go/src/runtime/panic.go:884 +0x213
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*trInvoker).Invoke(0xc0001dbce0?, {0x11f39f0?, 0xc0000415c0?}, {0xf8c160?, 0xc000011e60?}, 0xc0001dbce0?)
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/sdf_invokers.go:320 +0x1b3
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*TruncateSizedRestriction).ProcessElement(0xc000041540, {0x11f39f0, 0xc0000415c0}, 0xc0001dba40, {0x0, 0x0, 0x0})
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/sdf.go:345 +0x118
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).Process(0xc000497040, {0x11f39f0, 0xc0000415c0})
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/datasource.go:189 +0x510
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute.func2({0x11f39f0?, 0xc0000415c0?})
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:131 +0x42
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic({0x11f39f0?, 0xc0000415c0?}, 0x4172ed?)
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:62 +0x6c
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc0004b39e0, {0x11f39f0, 0xc0000415c0}, {0xc0001370f8, 0x13}, {{0x11ebaa0?, 0xc000111f20?}, {0x1208f98?, 0xc00038c7e0?}})
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:130 +0x3da
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc000258000, {0x11f38d8, 0xc0004d93e0}, 0xc000338370)
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:407 +0xab7
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main.func4({0x11f38d8, 0xc0004d93e0}, 0xc000338370)
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:193 +0x19d
created by github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main
/home/rru/temp/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:212 +0xfed
"
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142668196
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
Review Comment:
Looking at the "default" implementation which does the same, a `nil` is probably the best actually, since it will abort processing immeadiately.
The default implementation is to handle unbounded restrictions, but is sort of a little less obvious due to the Estimator involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142385497
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
Review Comment:
Thanks for elaborating on this. Would it be clearer to return the currently tracked restriction, `rt.GetRestriction().(offsetrange.Restriction)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142670336
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
(3) That's also very good. SGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142662399
##########
sdks/go/pkg/beam/transforms/periodic/periodic_test.go:
##########
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package periodic
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/testing/passert"
+)
+
+func TestSequence(t *testing.T) {
+ p, s := beam.NewPipelineWithRoot()
+ sd := SequenceDefinition{
+ Interval: time.Second,
+ Start: 0,
+ End: time.Minute.Milliseconds(),
+ }
+ in := beam.Create(s, sd)
+ out := Sequence(s, in)
+ passert.Count(s, out, "SecondsInMinute", 60)
+ beam.Init()
Review Comment:
Two things:
First, `beam.Init()` should be called before pipeline construction, since it's what enables the binary to behave as either a worker or a main.
Second, calling beam.Init in tests isn't recommended as a rule since it can/will get called multiple times.
So we have the PTest package to call it after test binary flag parsing, by overriding the default TestMain.
If you add the following:
```
func TestMain(m *testing.M) {
os.Exit(ptest.MainRetWithDefault(m, "prism"))
}
```
You can even then trigger pipeline execution with `ptest.RunAndValidate(t, p)`, which will fail if an error is returned, but also check any counters from the returned pipeline result, which is another way we could validate this behavior other than `passert.Count`.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
(2) Very simple: For impulse, add a DoFn that accepts the int64 and returns []byte. `func(_ int64, emit func([]byte)) { emit([]byte{}) } ` (Emitters are marginally more efficient than direct returns for DoFns.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464922673
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464935873
## [Codecov](https://codecov.io/gh/apache/beam/pull/25808?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#25808](https://codecov.io/gh/apache/beam/pull/25808?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a147b8) into [master](https://codecov.io/gh/apache/beam/commit/b1ea4d3554050d218dbf004908dba7a879daec90?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b1ea4d3) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #25808 +/- ##
=======================================
Coverage 72.80% 72.80%
=======================================
Files 775 775
Lines 102948 102948
=======================================
Hits 74956 74956
Misses 26537 26537
Partials 1455 1455
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `53.95% <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1464935714
Thanks for the contribution! This is a very useful transform. For the drain issue, Python and Java implementation also did not drain until recent fix (#23765 #25716). Basically the transform should (i) update watermark when advance and (ii) truncate immediately when truncate is called (happens in drain)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142383516
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
+
+ startTime, _ := time.Parse(time.RFC3339, periodicSequenceStart)
+ endTime, _ := time.Parse(time.RFC3339, periodicSequenceEnd)
+ sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
+ beam.ParDo(
+ s,
+ update,
+ periodic.Impulse(
+ s,
+ startTime,
+ endTime,
+ periodicSequenceInterval,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
WRT the cache:
Beam's execution abstraction, the FnAPI, has provisions for caching data cross bundle from the StateAPI on the SDK side, in order to avoid repeated deserialization, and additional round trips to the primary store of the runner to fetch state data. Side inputs also come across the StateAPI. In particular, very valuable for streaming jobs, as typically a single "SDK harness" is usually responsible for the same key all the time, so for tight windows it would look up the same data from the side inputs.
---
WRT not applying after the impulse change. Makes sense since an int64 was being received from the "update" function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142391555
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
Review Comment:
The direct runner (and basically all non-portable runners), are not great, since they mis-align expectations for users when they move to executing on a "real" runner, like Flink or Dataflow. There's also no "spec" about what a Direct Runner should do to properly test things, so the Go Direct Runner is missing a number of features that the Java and Python ones use.
Essentially Prism is going to replace the Go Direct Runner as the default runner for the Go SDK at least, and hopefully make testing and local runs for all facets of Beam easier. The prism Readme has a bunch of the vision, and desired goals, but I gave a talk at last year's Beam Summit, about the motivations, especially in all the "hidden" bits of beam that users usually don't interact with, but are affected by.
https://www.youtube.com/watch?v=G4lbkvAG6xk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1476634278
The drain panic is outside of your code. From [here]( https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf_invokers.go#L320) which means it's invoking a Drain for a KV for some reason... But sequenceGenDoFn isn't a KV, it does have restrictions though.
OK the thing that's failing here is that a recent change to support context.Context as a parameter into all the SDF methods changed how this one executes.
In particular, it moved from determining what to pass into the function based on its signature to based on what's input. But that's not how this Transform needs to work.
"old code"
https://github.com/apache/beam/blame/9fb9d2cb415e903e4222021f8848bf4420157b2a/sdks/go/pkg/beam/core/runtime/exec/sdf_invokers.go#L337
Ooof. This is why I want prism to implement all of Beam... including Drains, so this stuff can be tested without involving Dataflow.
--------
The fix is to change the trInvoker method a little, since it's generally passing in the wrong thing anyway. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L296
The problem we're running into is deciding whether the Main Input is a KV or not, so the output can be expanded out. This is something unique to the Go SDK since we didn't have generics to lean on in the original design to have a "real" KV.
Basically, we are seeing a *FullValue{ Elm: Input, Elm2: Restriction}.
So the right place to change is actually what's passed into the invoker in exec/sdf.go.
There's already handling code there... Let me think about this for a bit, since I need to follow what's happening in the code...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: Add periodic.Sequence and periodic.Impulse transforms
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1475327090
Thanks for your patience! I'm still getting caught up from being on vacation. I'm intending to get this review done today or tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142783779
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
Ah, of course, just flip it. I've made the change so that `Impulse` emits `[]byte` and `Sequence` emits the `currentOutputIndex int64`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1477108935
I believe that the last points raised should now be addressed. Please take one last pass when you have a chance, and a big thanks for your input and help on this- much appreciated :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1142401126
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
Review Comment:
While I agree in principle, if we were to take that route we should add a constructor function to the sdf package that does that instead `NewManualWatermarkEstimator()`, so it's clearer where to start, and avoid others needing to think about it as well.
On the the other hand, the Unix Epoch was 50+ years ago, and users that would be trying to reprocess data dated older than that would know they're probably going to need some special handling around it, and as you've said, it can be worked around already. I'd say, we do nothing at this juncture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hnnsgstfssn commented on a diff in pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "hnnsgstfssn (via GitHub)" <gi...@apache.org>.
hnnsgstfssn commented on code in PR #25808:
URL: https://github.com/apache/beam/pull/25808#discussion_r1133221543
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
Review Comment:
Was thinking, while this zero `time.Time` value seems to work, it might be clearer if it was initialized to `mtime.MinTimestamp.ToTime()`. What do you think?
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
Review Comment:
To satisfy the linter I've changed this to `mtime.Time(sd.End).ToTime().Sub(mtime.Time(sd.Start).ToTime()) / sd.Interval`.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
Review Comment:
Indeed I struggled with the direct runner, but have now added two working tests that uses the `prism` runner. I am struggling to understand how to fold this into the sequence definition as you suggest, to properly test it. Instead I went ahead and removed this entirely and the DoFn uses `time.Now`.
I am happy to leave it there for now. If you want to expand on how to fold it into the definition and test it I am happy to add this as well.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.now = now
+ return nil
+ }
+}
+
+// Impulse is a PTransform which generates a sequence of timestamped
+// elements at fixed runtime intervals. If [WithApplyWindow] is specified, each
+// element will be assigned to its own fixed window of interval size.
+//
+// The transform behaves the same as [Sequence] transform, but can be
+// used as the first transform in a pipeline.
+//
+// The following applies to the arguments.
+// - if interval <= 0, interval is set to [math.MaxInt64]
+// - if start is a zero value [time.Time], start is set to the current time
+// - if start is after end, start is set to end
+//
+// The PCollection generated by Impulse is unbounded and the output elements
+// are the [time.UnixMilli] int64 values of the output timestamp.
+func Impulse(s beam.Scope, start, end time.Time, interval time.Duration, opts ...ImpulseOption) beam.PCollection {
+ if interval <= 0 {
+ interval = math.MaxInt64
+ }
+ if start.IsZero() {
+ start = time.Now()
+ }
+ if start.After(end) {
+ start = end
+ }
Review Comment:
Thanks for sharing your thoughts here.
(1) Since we are not able to use `mtime.Time` directly, I agree we should instead use `int64`s to store the `time.UnixMilli` values on `SequenceDefinition`. I have made this change.
(2) I agree that it is odd to send the duplicate `int64` values and that emitting `[]byte` better aligns with the existing `beam.Impulse`. I also agree that it makes sense to output `int64` from the `Sequence` transform. I have changed `sequenceGenDoFn` to output `[]byte`, but the `Sequence` transform still shares the same implementation and will also output `[]byte`. Can you suggest a neat way to share the implementation but still return `int64` from the `Sequence` transform?
(3) I've slightly reworked the validation of start, end and interval. Start and end are normalized into `[mtime.MinTimestamp,mtime.MaxTimestamp]` using `mtime.Normalize` and the interval is set to `end.Sub(start)` if it is unset or invalid. I wasn't sure which way to go based on your feedback, but let me know what you think about this change.
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
+
+ startTime, _ := time.Parse(time.RFC3339, periodicSequenceStart)
+ endTime, _ := time.Parse(time.RFC3339, periodicSequenceEnd)
+ sideInput := beam.WindowInto(s, window.NewFixedWindows(periodicSequenceInterval),
+ beam.ParDo(
+ s,
+ update,
+ periodic.Impulse(
+ s,
+ startTime,
+ endTime,
+ periodicSequenceInterval,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
I thought of making the side input window larger. Do you think it is worth making that change, taking another configuration to specify the side input window size?
I'm also curious to know what you mean by
> this behavior isn't yet enabled by default in the Go SDK
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
Review Comment:
Sounds good to me! I've gone ahead and removed the functional options entirely in favour of a simple `applyWindow bool`. This also makes some of the other comments moot.
##########
sdks/go/examples/slowly_updating_side_input/slowly_updating_side_input.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package main
+
+import (
+ "context"
+ "flag"
+ "strings"
+ "time"
+
+ "cloud.google.com/go/pubsub"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window/trigger"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/options/gcpopts"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+ _ "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/periodic"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
+)
+
+func init() {
+ register.Function4x0(update)
+ register.Function4x0(process)
+ register.Emitter2[int, string]()
+ register.Iter1[string]()
+}
+
+// update simulates an external call to get data for the side input.
+func update(ctx context.Context, t beam.EventTime, i int64, emit func(int, string)) {
+ log.Infof(ctx, "Making external call %d at %s", i, t.ToTime().Format(time.RFC3339))
+
+ // zero is the key used in beam.AddFixedKey which will be applied on the main input.
+ id, externalData := 0, "some fake data that changed at "+time.Now().Format(time.RFC3339)
+
+ emit(id, externalData)
+}
+
+// process simulates processing of main input. It reads side input by key
+func process(ctx context.Context, k int, v []byte, side func(int) func(*string) bool) {
+ log.Infof(ctx, "Processing (key:%d,value:%q)", k, v)
+
+ iter := side(k)
+
+ var externalData []string
+ var externalDatum string
+ for iter(&externalDatum) {
+ externalData = append(externalData, externalDatum)
+ }
+
+ log.Infof(ctx, "Processing (key:%d,value:%q) with external data %q", k, v, strings.Join(externalData, ","))
+}
+
+func fatalf(err error, format string, args ...interface{}) {
+ if err != nil {
+ log.Fatalf(context.TODO(), format, args...)
+ }
+}
+
+func main() {
+ var inputTopic, periodicSequenceStart, periodicSequenceEnd string
+ var periodicSequenceInterval time.Duration
+
+ now := time.Now()
+
+ flag.StringVar(&periodicSequenceStart, "periodic_sequence_start", now.Add(-1*time.Hour).Format(time.RFC3339),
+ "The time at which to start the periodic sequence.")
+
+ flag.StringVar(&periodicSequenceEnd, "periodic_sequence_end", now.Add(100*time.Hour).Format(time.RFC3339),
+ "The time at which to end the periodic sequence.")
+
+ flag.DurationVar(&periodicSequenceInterval, "periodic_sequence_interval", 1*time.Minute,
+ "The interval between periodic sequence output.")
+
+ flag.StringVar(&inputTopic, "input_topic", "input",
+ "The PubSub topic from which to read the main input data.")
+
+ flag.Parse()
+ beam.Init()
+ ctx := context.Background()
+ p, s := beam.NewPipelineWithRoot()
+
+ project := gcpopts.GetProject(ctx)
+ client, err := pubsub.NewClient(ctx, project)
+ fatalf(err, "Failed to create client: %v", err)
+ _, err = pubsubx.EnsureTopic(ctx, client, inputTopic)
+ fatalf(err, "Failed to ensure topic: %v", err)
+
+ mainInput := beam.WindowInto(
+ s,
+ window.NewFixedWindows(periodicSequenceInterval),
+ beam.AddFixedKey( // simulate keyed data by adding a fixed key
+ s,
+ pubsubio.Read(
+ s,
+ project,
+ inputTopic,
+ nil,
+ ),
+ ),
+ beam.Trigger(trigger.Repeat(trigger.Always())),
+ beam.PanesDiscard(),
+ )
Review Comment:
Gotcha! I do agree and I'll keep it in mind for next time.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
+
+// WithApplyWindow configures the [Impulse] transform to apply a fixed window
+// transform to the output PCollection.
+func WithApplyWindow() ImpulseOption {
+ return func(o *impulseConfig) error {
+ o.ApplyWindow = true
+ return nil
+ }
+}
+
+func withNowFunc(now func() time.Time) ImpulseOption {
Review Comment:
Removed.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
Review Comment:
That's great! I couldn't get it working on the direct runner, but didn't try using the new prism explicitly. Have now added two tests `TestImpulse` and `TestSequence` and it looks like that's working. Let me know what you think.
##########
sdks/go/pkg/beam/transforms/periodic/periodic.go:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package periodic contains transformations for generating periodic sequences.
+package periodic
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "reflect"
+ "time"
+
+ "github.com/apache/beam/sdks/v2/go/pkg/beam"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange"
+ "github.com/apache/beam/sdks/v2/go/pkg/beam/register"
+)
+
+func init() {
+ register.DoFn5x2[context.Context, *sdf.ManualWatermarkEstimator, *sdf.LockRTracker, SequenceDefinition,
+ func(beam.EventTime, int64),
+ sdf.ProcessContinuation, error](&sequenceGenDoFn{})
+ register.Emitter2[beam.EventTime, int64]()
+ beam.RegisterType(reflect.TypeOf(SequenceDefinition{}))
+}
+
+// SequenceDefinition holds the configuration for generating a sequence of
+// timestamped elements at an interval.
+type SequenceDefinition struct {
+ Interval time.Duration
+ Start time.Time
+ End time.Time
+}
+
+type sequenceGenDoFn struct {
+ now func() time.Time
+}
+
+func (fn *sequenceGenDoFn) Setup() {
+ if fn.now == nil {
+ fn.now = time.Now
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateInitialRestriction(sd SequenceDefinition) offsetrange.Restriction {
+ totalOutputs := math.Ceil(float64(sd.End.Sub(sd.Start) / sd.Interval))
+ return offsetrange.Restriction{
+ Start: int64(0),
+ End: int64(totalOutputs),
+ }
+}
+
+func (fn *sequenceGenDoFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker {
+ return sdf.NewLockRTracker(offsetrange.NewTracker(rest))
+}
+
+func (fn *sequenceGenDoFn) RestrictionSize(_ SequenceDefinition, rest offsetrange.Restriction) float64 {
+ return rest.Size()
+}
+
+func (fn *sequenceGenDoFn) SplitRestriction(_ SequenceDefinition, rest offsetrange.Restriction) []offsetrange.Restriction {
+ return []offsetrange.Restriction{rest}
+}
+
+// TruncateRestriction immediately truncates the entire restrication.
+func (fn *sequenceGenDoFn) TruncateRestriction(_ *sdf.LockRTracker, _ SequenceDefinition) offsetrange.Restriction {
+ return offsetrange.Restriction{}
+}
+
+func (fn *sequenceGenDoFn) CreateWatermarkEstimator() *sdf.ManualWatermarkEstimator {
+ return &sdf.ManualWatermarkEstimator{}
+}
+
+func (fn *sequenceGenDoFn) ProcessElement(ctx context.Context, we *sdf.ManualWatermarkEstimator, rt *sdf.LockRTracker, sd SequenceDefinition, emit func(beam.EventTime, int64)) (sdf.ProcessContinuation, error) {
+ currentOutputIndex := rt.GetRestriction().(offsetrange.Restriction).Start
+ currentOutputTimestamp := sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime := fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ for currentOutputTimestamp.Before(currentTime) {
+ if rt.TryClaim(currentOutputIndex) {
+ emit(mtime.FromTime(currentOutputTimestamp), currentOutputTimestamp.UnixMilli())
+ currentOutputIndex += 1
+ currentOutputTimestamp = sd.Start.Add(sd.Interval * time.Duration(currentOutputIndex))
+ currentTime = fn.now()
+ we.UpdateWatermark(currentOutputTimestamp)
+ } else if err := rt.GetError(); err != nil || rt.IsDone() {
+ // Stop processing on error or completion
+ return sdf.StopProcessing(), rt.GetError()
+ } else {
+ return sdf.ResumeProcessingIn(sd.Interval), nil
+ }
+ }
+
+ return sdf.ResumeProcessingIn(time.Until(currentOutputTimestamp)), nil
+}
+
+type impulseConfig struct {
+ ApplyWindow bool
+
+ now func() time.Time
+}
+
+type impulseOption func(*impulseConfig) error
+
+// ImpulseOption is a function that configures an [Impulse] transform.
+type ImpulseOption = impulseOption
Review Comment:
Since we've opted for using `applyWindow bool` this has now been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #25808: [#23106] Add periodic.Sequence and periodic.Impulse transforms to Go SDK
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #25808:
URL: https://github.com/apache/beam/pull/25808#issuecomment-1476926631
It's very pleasing that we can get coverage for this! https://app.codecov.io/gh/apache/beam/pull/25808
Can't help on the drain side yet, but we'll get there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org