You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by tv...@apache.org on 2025/07/29 20:49:13 UTC
(beam) branch master updated: Add documentation for Wait.On and WaitOn transforms (#35647)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c52993ebb62 Add documentation for Wait.On and WaitOn transforms (#35647)
c52993ebb62 is described below
commit c52993ebb625db9b6317ba7dd31dd7c21f154745
Author: Shehab <12...@users.noreply.github.com>
AuthorDate: Tue Jul 29 23:49:06 2025 +0300
Add documentation for Wait.On and WaitOn transforms (#35647)
* Add documentation for Wait.On and WaitOn transforms
* Add descriptive comments to documentation
* Enhance documentation for Wait.On transform and add links in the navigation menu
* Add JavaDoc link, Update java examples to use consistent terminology
* Modify examples for compatibility with Beam Playground
* Update Python Examples for Beam Playground compatibility
* Update Java Wait.On Example for consistency
* Fix Linting issues
* Fix indentation in Python WaitOn example
* Fix indentation in WaitOn example 2
* Fix indentation in Java Examples
---
.../en/documentation/transforms/java/other/wait.md | 106 +++++++++++++++++++++
.../en/documentation/transforms/java/overview.md | 3 +-
.../transforms/python/other/waiton.md | 61 ++++++++++++
.../en/documentation/transforms/python/overview.md | 2 +
.../partials/section-menu/en/documentation.html | 2 +
5 files changed, 173 insertions(+), 1 deletion(-)
diff --git a/website/www/site/content/en/documentation/transforms/java/other/wait.md b/website/www/site/content/en/documentation/transforms/java/other/wait.md
new file mode 100644
index 00000000000..ad28e3d3090
--- /dev/null
+++ b/website/www/site/content/en/documentation/transforms/java/other/wait.md
@@ -0,0 +1,106 @@
+---
+title: "Wait.On"
+---
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Wait.On
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Wait.html">
+ <img src="/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br><br>
+
+`Wait.On` returns a `PCollection` with the contents identical to the input `PCollection`, but delays the downstream processing until one or more other `PCollections` (signals) have finished processing. This is useful for enforcing ordering or dependencies between different parts of a pipeline, especially when some outputs interact with external systems (such as writing to a database).
+
+When you apply `Wait.On`, the elements of the main `PCollection` will not be emitted for downstream processing until the computations required to produce the specified signal `PCollections` have completed. In streaming mode, this is enforced per window: the corresponding window of each waited-on `PCollection` must close before elements are passed through.
+
+## Examples
+**Example 1**: Basic usage
+
+{{< highlight java >}}
+PipelineOptions options = PipelineOptionsFactory.create();
+Pipeline p = Pipeline.create(options);
+PCollection<String> main = p.apply("CreateMain", Create.of("item1", "item2", "item3"));
+PCollection<Void> signal = p.apply("CreateSignal", Create.of("trigger"))
+ .apply("ProcessSignal", ParDo.of(new DoFn<String, Void>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws InterruptedException {
+ // Simulate some processing time
+ Thread.sleep(2000);
+ // Signal processing complete
+ }
+}));
+// Wait for 'signal' to complete before processing 'main'
+// Elements pass through unchanged after 'signal' finishes
+PCollection<String> processed = main.apply("WaitOnSignal", Wait.on(signal))
+ .apply("ProcessAfterWait", MapElements.into(TypeDescriptors.strings())
+ .via(item -> "Processed: " + item));
+processed.apply("LogResults", ParDo.of(new DoFn<String, Void>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ System.out.println(c.element());
+ }
+}));
+{{< /highlight >}}
+
+**Example 2**: Using multiple signals
+
+{{< highlight java >}}
+PipelineOptions options = PipelineOptionsFactory.create();
+Pipeline p = Pipeline.create(options);
+// The PCollection to be processed after the signals.
+PCollection<String> main2 = p.apply("CreateMain2", Create.of("data1", "data2"));
+// Signal 1: Simulate a long-running setup task.
+PCollection<Void> signal1 = p.apply("CreateSignal1", Create.of("setup"))
+ .apply("SetupDatabase", ParDo.of(new DoFn<String, Void>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws InterruptedException {
+ System.out.println("Starting database setup...");
+ Thread.sleep(1000);
+ System.out.println("Database setup complete.");
+ }
+ }));
+// Signal 2: Simulate loading a configuration file.
+PCollection<Void> signal2 = p.apply("CreateSignal2", Create.of("config"))
+ .apply("LoadConfig", ParDo.of(new DoFn<String, Void>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws InterruptedException {
+ System.out.println("Loading configuration...");
+ Thread.sleep(1500);
+ System.out.println("Configuration loaded.");
+ }
+ }));
+// Wait for both signal1 and signal2 to complete before processing main2.
+PCollection<String> result2 = main2.apply("WaitOnSignals", Wait.on(signal1, signal2))
+ .apply("TransformData", MapElements.into(TypeDescriptors.strings())
+ .via(data -> data.toUpperCase() + "_READY"));
+// Log the final results.
+result2.apply("LogResults", ParDo.of(new DoFn<String, Void>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ System.out.println("Final Result: " + c.element());
+ }
+}));
+{{< /highlight >}}
+
+## Related transforms
+* [Flatten](/documentation/transforms/java/other/flatten) merges multiple `PCollection` objects into a single logical `PCollection`.
+* [Window](/documentation/transforms/java/other/window) logically divides or groups elements into finite windows.
+* [WithTimestamps](/documentation/transforms/java/elementwise/withtimestamps) assigns timestamps to elements in a collection.
diff --git a/website/www/site/content/en/documentation/transforms/java/overview.md b/website/www/site/content/en/documentation/transforms/java/overview.md
index 63a9797b56a..59aa93930fb 100644
--- a/website/www/site/content/en/documentation/transforms/java/overview.md
+++ b/website/www/site/content/en/documentation/transforms/java/overview.md
@@ -76,4 +76,5 @@ limitations under the License.
<tr><td><a href="/documentation/transforms/java/other/view">View</a></td><td>Operations for turning a collection into view that may be used as a side-input to a <code>ParDo</code>.</td></tr>
<tr><td><a href="/documentation/transforms/java/other/window">Window</a></td><td>Logically divides up or groups the elements of a collection into finite
windows according to a provided <code>WindowFn</code>.</td></tr>
-</table>
+ <tr><td><a href="/documentation/transforms/java/other/wait">Wait</a></td><td>Delays processing of a PCollection until other PCollections have finished processing.</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/www/site/content/en/documentation/transforms/python/other/waiton.md b/website/www/site/content/en/documentation/transforms/python/other/waiton.md
new file mode 100644
index 00000000000..8cdf8a47ad4
--- /dev/null
+++ b/website/www/site/content/en/documentation/transforms/python/other/waiton.md
@@ -0,0 +1,61 @@
+---
+title: "WaitOn"
+---
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WaitOn
+
+`WaitOn` returns a `PCollection` with the contents identical to the input `PCollection`, but delays the downstream processing until one or more other `PCollections` (signals) have finished processing. This is useful for enforcing ordering or dependencies between different parts of a pipeline, especially when some outputs interact with external systems (such as writing to a database).
+
+When you apply `WaitOn`, the elements of the main `PCollection` will not be emitted for downstream processing until the computations required to produce the specified signal `PCollections` have completed. In streaming mode, this is enforced per window: the corresponding window of each waited-on `PCollection` must close before elements are passed through.
+
+## Examples
+
+```python
+import time
+import apache_beam as beam
+from apache_beam.transforms.util import WaitOn
+
+# Example 1: Basic usage
+with beam.Pipeline(options=pipeline_options) as p:
+ main = p | 'CreateMain' >> beam.Create([1, 2, 3])
+ signal = (
+ p | 'CreateSignal' >> beam.Create(['a', 'b'])
+ | 'ProcessSignal' >> beam.Map(lambda x: print(f"Processing signal element: {x}") or time.sleep(2)))
+ # Wait for 'signal' to complete before processing 'main'
+ result = main | 'WaitOnSignal' >> WaitOn(signal)
+ # Print each result to logs.
+ result | 'PrintExample1' >> beam.Map(lambda x: print(f"Example 1 Final Output: {x}"))
+
+# Example 2: Using multiple signals
+with beam.Pipeline(options=pipeline_options) as p:
+ main = p | 'CreateMain' >> beam.Create(['item1', 'item2', 'item3'])
+ signal1 = (
+ p | 'CreateSignal_A' >> beam.Create(['setup_db'])
+ | 'ProcessSignal_A' >> beam.Map(lambda x: print("Signal A: Setting up database...") or time.sleep(1)))
+ signal2 = (
+ p | 'CreateSignal_B' >> beam.Create(['load_config'])
+ | 'ProcessSignal_B' >> beam.Map(lambda x: print("Signal B: Loading config...") or time.sleep(3)))
+ # Wait for both 'signal1' and 'signal2' to complete before processing 'main'
+ result = main | 'WaitOnSignals' >> WaitOn(signal1, signal2)
+ # Print each result to logs.
+ result | 'PrintExample2' >> beam.Map(lambda x: print(f"Example 2 Final Output: {x.upper()}_READY"))
+```
+
+## Related transforms
+* [Flatten](/documentation/transforms/python/other/flatten) merges multiple `PCollection` objects into a single logical `PCollection`.
+* [WindowInto](/documentation/transforms/python/other/windowinto) logically divides or groups elements into finite windows.
+* [Reshuffle](/documentation/transforms/python/other/reshuffle) redistributes elements between workers.
diff --git a/website/www/site/content/en/documentation/transforms/python/overview.md b/website/www/site/content/en/documentation/transforms/python/overview.md
index 34599ed6c02..c51af84b1a8 100644
--- a/website/www/site/content/en/documentation/transforms/python/overview.md
+++ b/website/www/site/content/en/documentation/transforms/python/overview.md
@@ -82,4 +82,6 @@ limitations under the License.
most useful for adjusting parallelism or preventing coupled failures.</td></tr>
<tr><td><a href="/documentation/transforms/python/other/windowinto">WindowInto</a></td><td>Logically divides up or groups the elements of a collection into finite
windows according to a function.</td></tr>
+ <tr><td><a href="/documentation/transforms/python/other/waiton">WaitOn</a></td><td>Delays processing of a PCollection until other PCollections have finished processing.</td></tr>
</table>
+
diff --git a/website/www/site/layouts/partials/section-menu/en/documentation.html b/website/www/site/layouts/partials/section-menu/en/documentation.html
index 6b37450786f..2386ecb39d9 100755
--- a/website/www/site/layouts/partials/section-menu/en/documentation.html
+++ b/website/www/site/layouts/partials/section-menu/en/documentation.html
@@ -356,6 +356,7 @@
<li><a href="/documentation/transforms/python/other/create/">Create</a></li>
<li><a href="/documentation/transforms/python/other/flatten/">Flatten</a></li>
<li><a href="/documentation/transforms/python/other/reshuffle/">Reshuffle</a></li>
+ <li><a href="/documentation/transforms/python/other/waiton/">WaitOn</a></li>
<li><a href="/documentation/transforms/python/other/windowinto/">WindowInto</a></li>
</ul>
</li>
@@ -417,6 +418,7 @@
<li><a href="/documentation/transforms/java/other/flatten/">Flatten</a></li>
<li><a href="/documentation/transforms/java/other/passert/">PAssert</a></li>
<li><a href="/documentation/transforms/java/other/view/">View</a></li>
+ <li><a href="/documentation/transforms/java/other/wait/">Wait.On</a></li>
<li><a href="/documentation/transforms/java/other/window/">Window</a></li>
</ul>
</li>