You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/14 18:10:54 UTC
[GitHub] [beam] Ardagan opened a new pull request #11415: [BEAM-9650]
Cleanup documentation on side inputs patterns
Ardagan opened a new pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408434733
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
+period and SDF Read operation or similar to read data into side input
+periodically:
1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
1. Read data using Read operation triggered by arrival of PCollection element.
Review comment:
On a second look, I think it's redundant to say "Read data using a Read operation" - maybe we can say something like "Read data when a PCollection element arrives"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408980179
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
Review comment:
I might be missing something, but that's exactly the phrasing. "PTransforms" word is on next line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408358586
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,24 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
Review comment:
In Beam, we have IO.Read transforms that do read source data. Later data read can be assigned to specific window.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408951043
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
+PTransforms:
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
+1. Use the PeriodicImpulse or PeriodicSequence PTransform to generate an
+infinite sequence of elements at required processing time intervals and assign
+them to separate windows.
1. Read data using Read operation triggered by arrival of PCollection element.
Review comment:
I see you marked the previous comment here as resolved but I don't see any changes to this line. Was that intended?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408431325
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
Review comment:
It's probably best to replace "To do this" with "To do ABC" - it'll help the reader figure out what "this" refers to. Otherwise, if the reader is just skimming the page and starts at this paragraph, they have to read the previous paragraph to figure out what the pronoun refers to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408999158
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,34 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
-
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
-1. Read data using Read operation triggered by arrival of PCollection element.
-
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
+
+Implementation of described approach can be narrowed down to:
+
+1. Use the PeriodicImpulse or PeriodicSequence PTransform to:
+ * Generate an infinite sequence of elements at required processing time
+ intervals
+ * Assign them to separate windows.
+1. Fetch data using SDF Read or ReadAll PTransform triggered by arrival of
+PCollection element.
1. Apply side input.
Review comment:
Missing "the" -> "Apply the side input"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408448451
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
+period and SDF Read operation or similar to read data into side input
+periodically:
1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
1. Read data using Read operation triggered by arrival of PCollection element.
1. Apply side input.
-```python
+```py
Review comment:
Seems that adding this snippet makes java snippet at L45 disappear from served site despite me not changing anything in code above. Do you know if there's some trick to make both snippets appear?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on issue #11415: [BEAM-9650] Cleanup
documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on issue #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#issuecomment-613597087
R: @soyrice
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408942009
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
+PTransforms:
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
+1. Use the PeriodicImpulse or PeriodicSequence PTransform to generate an
Review comment:
Oh! Can we format this with bullet points:
1. Use the PeriodicImpulse... to:
* Generate an infinite sequence of elements...
* Assign the elements to separate windows...
1. Read data...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408430781
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
Review comment:
Missing "the" -> "on the main input"
Present tense -> "is matched to a single..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r409124997
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
Review comment:
I probably just missed the word. Looks good in staging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408943766
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
Review comment:
How about something like this: "To read side input data periodically... use a combination of PeriodicSequence..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408994489
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
Review comment:
Gotcha - the intent is to summarize the previous abstract, so that it's easier for readers to skim the doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408998445
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,34 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
-
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
-1. Read data using Read operation triggered by arrival of PCollection element.
-
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
+
+Implementation of described approach can be narrowed down to:
Review comment:
We should rephrase as "To read side input data periodically:"
The goal is to summarize the previous paragraph, so that it's clear what these steps refer to without needing to read through the previous paragraph. The previous paragraph offers additional context, but if a reader skims the page, they should be able to quickly figure out what these steps are about.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on issue #11415: [BEAM-9650] Cleanup
documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on issue #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#issuecomment-614275077
LGTM (modulo one last comment). Thanks again for opening this PR and doing an editorial review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408433930
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
+period and SDF Read operation or similar to read data into side input
Review comment:
I think we should split this sentence into two - it's hard to follow. A good place to start a new sentence is at "SDF Read operation"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408990151
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
Review comment:
I'm a bit confused here since adding text you suggest is pretty much re-writing same thing that is a header and previous abstract. In next iteration I tried to remove this abstract overall since list of steps provides same information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on issue #11415: [BEAM-9650] Cleanup
documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on issue #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#issuecomment-614164320
> The text LGTM after resolving the remaining comments, but I don't see the code sample on the staging server.
I think it's missing because it's not picking up the language selector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408432853
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
Review comment:
For formatting, it might be clearer to say "a combination of PeriodicSequence and PeriodicImpulse PTransforms" instead of "PeriodicSequence/PeriodicImpulse PTransforms" - just to make sure there's no ambiguity in regards to these being two separate transforms.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408433143
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
Review comment:
Present tense -> "PTransforms that generate"
Missing article -> "generate an infinite sequence..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408945442
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
Review comment:
SDF Read and ReadAll are transforms too, right? The following will help clarify what the combination consists of: "...PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll PTransforms"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408430834
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
Review comment:
What does the term "main input side" mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r409127336
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,34 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
-
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
Review comment:
I think we should remove "Later" because this is a part of the overall workflow described in the previous sentence, rather than a secondary step.
So this should be: "When you apply the side input to your main input..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] soyrice commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
soyrice commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408995787
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
+period and SDF Read operation or similar to read data into side input
+periodically:
1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
1. Read data using Read operation triggered by arrival of PCollection element.
1. Apply side input.
-```python
+```py
Review comment:
Yeah, the issue is that the page can only show one language at a time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan merged pull request #11415: [BEAM-9650] Cleanup
documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan merged pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408990367
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -46,29 +54,36 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
%}
```
+```py
+No sample present.
+```
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees consistency on the duration of the single window,
+meaning that each window on the main input will be matched to a single
+version of side input data.
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+Described approach can be implemented using combination of
+PeriodicSequence or PeriodicImpulse PTransforms and SDF Read or ReadAll
+PTransforms:
-1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
-
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
+1. Use the PeriodicImpulse or PeriodicSequence PTransform to generate an
+infinite sequence of elements at required processing time intervals and assign
+them to separate windows.
1. Read data using Read operation triggered by arrival of PCollection element.
Review comment:
Was not intended. FIxed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on issue #11415: [BEAM-9650] Cleanup
documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on issue #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#issuecomment-614167649
Staging job failing together with website publish jobs :( So staging version has old code unfortunately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan edited a comment on issue #11415: [BEAM-9650]
Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on issue #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#issuecomment-613597087
R: @soyrice @rosetn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Ardagan commented on a change in pull request #11415:
[BEAM-9650] Cleanup documentation on side inputs patterns
Posted by GitBox <gi...@apache.org>.
Ardagan commented on a change in pull request #11415: [BEAM-9650] Cleanup documentation on side inputs patterns
URL: https://github.com/apache/beam/pull/11415#discussion_r408453789
##########
File path: website/src/documentation/patterns/side-inputs.md
##########
@@ -50,25 +50,25 @@ For instance, the following code sample uses a `Map` to create a `DoFn`. The `Ma
## Slowly updating side input using windowing
-You can read side input pcollection periodically into distinct windows.
-Later, when you apply side input to your main input, windows will be matched automatically 1:1.
-This way, you can guarantee side input consistency on the duration of the single window.
-
-To do this, you can utilize PeriodicSequence PTransform that will generate infinite sequence
-of elements with some real-time period:
+You can read side input data periodically into distinct PCollection windows.
+Later, when you apply the side input to your main input, each main input
+window is automatically matched to a single side input window.
+This guarantees side input consistency on the duration of the single window,
+meaning that each window on main input side will be matched to a single
+version of side input data.
+
+To do this, you can utilize a combination of PeriodicSequence/PeriodicImpulse
+PTransforms that will generate infinite sequence of elements with some real-time
+period and SDF Read operation or similar to read data into side input
+periodically:
1. Use the PeriodicImpulse transform to generate windowed periodic sequence.
- a. MAX_TIMESTAMP can be replaced with some closer boundary if you want to stop generating elements at some point.
-
1. Read data using Read operation triggered by arrival of PCollection element.
1. Apply side input.
-```python
+```py
Review comment:
Website sets config for the page to show only python or java snippets. It seem to pick the last language provided as default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services