You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by tv...@apache.org on 2025/07/29 20:49:13 UTC

(beam) branch master updated: Add documentation for Wait.On and WaitOn transforms (#35647)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new c52993ebb62 Add documentation for Wait.On and WaitOn transforms (#35647)
c52993ebb62 is described below

commit c52993ebb625db9b6317ba7dd31dd7c21f154745
Author: Shehab <12...@users.noreply.github.com>
AuthorDate: Tue Jul 29 23:49:06 2025 +0300

    Add documentation for Wait.On and WaitOn transforms (#35647)
    
    * Add documentation for Wait.On and WaitOn transforms
    
    * Add descriptive comments to documentation
    
    * Enhance documentation for Wait.On transform and add links in the navigation menu
    
    * Add JavaDoc link, Update java examples to use consistent terminology
    
    * Modify examples for compatibility with Beam Playground
    
    * Update Python Examples for Beam Playground compatibility
    
    * Update Java Wait.On Example for consistency
    
    * Fix Linting issues
    
    * Fix indentation in Python WaitOn example
    
    * Fix indentation in WaitOn example 2
    
    * Fix indentation in Java Examples
---
 .../en/documentation/transforms/java/other/wait.md | 106 +++++++++++++++++++++
 .../en/documentation/transforms/java/overview.md   |   3 +-
 .../transforms/python/other/waiton.md              |  61 ++++++++++++
 .../en/documentation/transforms/python/overview.md |   2 +
 .../partials/section-menu/en/documentation.html    |   2 +
 5 files changed, 173 insertions(+), 1 deletion(-)

diff --git a/website/www/site/content/en/documentation/transforms/java/other/wait.md b/website/www/site/content/en/documentation/transforms/java/other/wait.md
new file mode 100644
index 00000000000..ad28e3d3090
--- /dev/null
+++ b/website/www/site/content/en/documentation/transforms/java/other/wait.md
@@ -0,0 +1,106 @@
+---
+title: "Wait.On"
+---
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Wait.On
+<table align="left">
+    <a target="_blank" class="button"
+        href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Wait.html">
+      <img src="/images/logos/sdks/java.png" width="20px" height="20px"
+           alt="Javadoc" />
+     Javadoc
+    </a>
+</table>
+<br><br>
+
+`Wait.On` returns a `PCollection` with the contents identical to the input `PCollection`, but delays the downstream processing until one or more other `PCollections` (signals) have finished processing. This is useful for enforcing ordering or dependencies between different parts of a pipeline, especially when some outputs interact with external systems (such as writing to a database).
+
+When you apply `Wait.On`, the elements of the main `PCollection` will not be emitted for downstream processing until the computations required to produce the specified signal `PCollections` have completed. In streaming mode, this is enforced per window: the corresponding window of each waited-on `PCollection` must close before elements are passed through.
+
+## Examples
+**Example 1**: Basic usage
+
+{{< highlight java >}}
+PipelineOptions options = PipelineOptionsFactory.create();
+Pipeline p = Pipeline.create(options);
+PCollection<String> main = p.apply("CreateMain", Create.of("item1", "item2", "item3"));
+PCollection<Void> signal = p.apply("CreateSignal", Create.of("trigger"))
+    .apply("ProcessSignal", ParDo.of(new DoFn<String, Void>() {
+        @ProcessElement
+        public void processElement(ProcessContext c) throws InterruptedException {
+            // Simulate some processing time
+            Thread.sleep(2000);
+            // Signal processing complete
+        }
+}));
+// Wait for 'signal' to complete before processing 'main'
+// Elements pass through unchanged after 'signal' finishes
+PCollection<String> processed = main.apply("WaitOnSignal", Wait.on(signal))
+    .apply("ProcessAfterWait", MapElements.into(TypeDescriptors.strings())
+        .via(item -> "Processed: " + item));
+processed.apply("LogResults", ParDo.of(new DoFn<String, Void>() {
+    @ProcessElement
+    public void processElement(ProcessContext c) {
+        System.out.println(c.element());
+    }
+}));
+{{< /highlight >}}
+
+**Example 2**: Using multiple signals
+
+{{< highlight java >}}
+PipelineOptions options = PipelineOptionsFactory.create();
+Pipeline p = Pipeline.create(options);
+// The PCollection to be processed after the signals.
+PCollection<String> main2 = p.apply("CreateMain2", Create.of("data1", "data2"));
+// Signal 1: Simulate a long-running setup task.
+PCollection<Void> signal1 = p.apply("CreateSignal1", Create.of("setup"))
+    .apply("SetupDatabase", ParDo.of(new DoFn<String, Void>() {
+      @ProcessElement
+      public void processElement(ProcessContext c) throws InterruptedException {
+        System.out.println("Starting database setup...");
+        Thread.sleep(1000);
+        System.out.println("Database setup complete.");
+      }
+    }));
+// Signal 2: Simulate loading a configuration file.
+PCollection<Void> signal2 = p.apply("CreateSignal2", Create.of("config"))
+    .apply("LoadConfig", ParDo.of(new DoFn<String, Void>() {
+      @ProcessElement
+      public void processElement(ProcessContext c) throws InterruptedException {
+        System.out.println("Loading configuration...");
+        Thread.sleep(1500);
+        System.out.println("Configuration loaded.");
+      }
+    }));
+// Wait for both signal1 and signal2 to complete before processing main2.
+PCollection<String> result2 = main2.apply("WaitOnSignals", Wait.on(signal1, signal2))
+    .apply("TransformData", MapElements.into(TypeDescriptors.strings())
+        .via(data -> data.toUpperCase() + "_READY"));
+// Log the final results.
+result2.apply("LogResults", ParDo.of(new DoFn<String, Void>() {
+    @ProcessElement
+    public void processElement(ProcessContext c) {
+        System.out.println("Final Result: " + c.element());
+    }
+}));
+{{< /highlight >}}
+
+## Related transforms
+* [Flatten](/documentation/transforms/java/other/flatten) merges multiple `PCollection` objects into a single logical `PCollection`.
+* [Window](/documentation/transforms/java/other/window) logically divides or groups elements into finite windows.
+* [WithTimestamps](/documentation/transforms/java/elementwise/withtimestamps) assigns timestamps to elements in a collection.
diff --git a/website/www/site/content/en/documentation/transforms/java/overview.md b/website/www/site/content/en/documentation/transforms/java/overview.md
index 63a9797b56a..59aa93930fb 100644
--- a/website/www/site/content/en/documentation/transforms/java/overview.md
+++ b/website/www/site/content/en/documentation/transforms/java/overview.md
@@ -76,4 +76,5 @@ limitations under the License.
   <tr><td><a href="/documentation/transforms/java/other/view">View</a></td><td>Operations for turning a collection into view that may be used as a side-input to a <code>ParDo</code>.</td></tr>
   <tr><td><a href="/documentation/transforms/java/other/window">Window</a></td><td>Logically divides up or groups the elements of a collection into finite
   windows according to a provided <code>WindowFn</code>.</td></tr>
-</table>
+  <tr><td><a href="/documentation/transforms/java/other/wait">Wait</a></td><td>Delays processing of a PCollection until other PCollections have finished processing.</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/www/site/content/en/documentation/transforms/python/other/waiton.md b/website/www/site/content/en/documentation/transforms/python/other/waiton.md
new file mode 100644
index 00000000000..8cdf8a47ad4
--- /dev/null
+++ b/website/www/site/content/en/documentation/transforms/python/other/waiton.md
@@ -0,0 +1,61 @@
+---
+title: "WaitOn"
+---
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WaitOn
+
+`WaitOn` returns a `PCollection` with the contents identical to the input `PCollection`, but delays the downstream processing until one or more other `PCollections` (signals) have finished processing. This is useful for enforcing ordering or dependencies between different parts of a pipeline, especially when some outputs interact with external systems (such as writing to a database).
+
+When you apply `WaitOn`, the elements of the main `PCollection` will not be emitted for downstream processing until the computations required to produce the specified signal `PCollections` have completed. In streaming mode, this is enforced per window: the corresponding window of each waited-on `PCollection` must close before elements are passed through.
+
+## Examples
+
+```python
+import time
+import apache_beam as beam
+from apache_beam.transforms.util import WaitOn
+
+# Example 1: Basic usage
+with beam.Pipeline(options=pipeline_options) as p:
+  main = p | 'CreateMain' >> beam.Create([1, 2, 3])
+  signal = (
+      p | 'CreateSignal' >> beam.Create(['a', 'b'])
+      | 'ProcessSignal' >> beam.Map(lambda x: print(f"Processing signal element: {x}") or time.sleep(2)))
+  # Wait for 'signal' to complete before processing 'main'
+  result = main | 'WaitOnSignal' >> WaitOn(signal)
+  # Print each result to logs.
+  result | 'PrintExample1' >> beam.Map(lambda x: print(f"Example 1 Final Output: {x}"))
+
+# Example 2: Using multiple signals
+with beam.Pipeline(options=pipeline_options) as p:
+  main = p | 'CreateMain' >> beam.Create(['item1', 'item2', 'item3'])
+  signal1 = (
+      p | 'CreateSignal_A' >> beam.Create(['setup_db'])
+      | 'ProcessSignal_A' >> beam.Map(lambda x: print("Signal A: Setting up database...") or time.sleep(1)))
+  signal2 = (
+      p | 'CreateSignal_B' >> beam.Create(['load_config'])
+      | 'ProcessSignal_B' >> beam.Map(lambda x: print("Signal B: Loading config...") or time.sleep(3)))
+  # Wait for both 'signal1' and 'signal2' to complete before processing 'main'
+  result = main | 'WaitOnSignals' >> WaitOn(signal1, signal2)
+  # Print each result to logs.
+  result | 'PrintExample2' >> beam.Map(lambda x: print(f"Example 2 Final Output: {x.upper()}_READY"))
+```
+
+## Related transforms
+* [Flatten](/documentation/transforms/python/other/flatten) merges multiple `PCollection` objects into a single logical `PCollection`.
+* [WindowInto](/documentation/transforms/python/other/windowinto) logically divides or groups elements into finite windows.
+* [Reshuffle](/documentation/transforms/python/other/reshuffle) redistributes elements between workers.
diff --git a/website/www/site/content/en/documentation/transforms/python/overview.md b/website/www/site/content/en/documentation/transforms/python/overview.md
index 34599ed6c02..c51af84b1a8 100644
--- a/website/www/site/content/en/documentation/transforms/python/overview.md
+++ b/website/www/site/content/en/documentation/transforms/python/overview.md
@@ -82,4 +82,6 @@ limitations under the License.
   most useful for adjusting parallelism or preventing coupled failures.</td></tr>
   <tr><td><a href="/documentation/transforms/python/other/windowinto">WindowInto</a></td><td>Logically divides up or groups the elements of a collection into finite
   windows according to a function.</td></tr>
+  <tr><td><a href="/documentation/transforms/python/other/waiton">WaitOn</a></td><td>Delays processing of a PCollection until other PCollections have finished processing.</td></tr>
 </table>
+
diff --git a/website/www/site/layouts/partials/section-menu/en/documentation.html b/website/www/site/layouts/partials/section-menu/en/documentation.html
index 6b37450786f..2386ecb39d9 100755
--- a/website/www/site/layouts/partials/section-menu/en/documentation.html
+++ b/website/www/site/layouts/partials/section-menu/en/documentation.html
@@ -356,6 +356,7 @@
             <li><a href="/documentation/transforms/python/other/create/">Create</a></li>
             <li><a href="/documentation/transforms/python/other/flatten/">Flatten</a></li>
             <li><a href="/documentation/transforms/python/other/reshuffle/">Reshuffle</a></li>
+            <li><a href="/documentation/transforms/python/other/waiton/">WaitOn</a></li>
             <li><a href="/documentation/transforms/python/other/windowinto/">WindowInto</a></li>
           </ul>
         </li>
@@ -417,6 +418,7 @@
             <li><a href="/documentation/transforms/java/other/flatten/">Flatten</a></li>
             <li><a href="/documentation/transforms/java/other/passert/">PAssert</a></li>
             <li><a href="/documentation/transforms/java/other/view/">View</a></li>
+            <li><a href="/documentation/transforms/java/other/wait/">Wait.On</a></li>
             <li><a href="/documentation/transforms/java/other/window/">Window</a></li>
           </ul>
         </li>