You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/10 05:21:24 UTC

[GitHub] [beam] leiyiz opened a new pull request #12807: [BEAM-2855] implement query 10

leiyiz opened a new pull request #12807:
URL: https://github.com/apache/beam/pull/12807


   implemented query 10, now query 10 is still not runnable because it seems like Beam team hasn't made up mind about what to do with this query. ([here](https://github.com/apache/beam/blob/master/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java#L122) is the comment in java query 10)
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12807      +/-   ##
   ==========================================
   - Coverage   40.38%   40.34%   -0.04%     
   ==========================================
     Files         451      452       +1     
     Lines       53721    53774      +53     
   ==========================================
     Hits        21696    21696              
   - Misses      32025    32078      +53     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...ce256db](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689990552


   R: @y1chi 
   R: @pabloem 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `41.91%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12807       +/-   ##
   ===========================================
   + Coverage   40.38%   82.30%   +41.91%     
   ===========================================
     Files         451      452        +1     
     Lines       53721    53803       +82     
   ===========================================
   + Hits        21696    44280    +22584     
   + Misses      32025     9523    -22502     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | [...runners/interactive/options/interactive\_options.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2ludGVyYWN0aXZlX29wdGlvbnMucHk=) | `86.66% <0.00%> (+6.66%)` | :arrow_up: |
   | ... and [270 more](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...0c716e2](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem merged pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12807:
URL: https://github.com/apache/beam/pull/12807


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] y1chi commented on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-692203332


   @pabloem Pablo could you help merge this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12807      +/-   ##
   ==========================================
   - Coverage   40.38%   40.34%   -0.04%     
   ==========================================
     Files         451      452       +1     
     Lines       53721    53774      +53     
   ==========================================
     Hits        21696    21696              
   - Misses      32025    32078      +53     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...0c716e2](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on a change in pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12807:
URL: https://github.com/apache/beam/pull/12807#discussion_r486742699



##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on a change in pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12807:
URL: https://github.com/apache/beam/pull/12807#discussion_r486742699



##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):

Review comment:
       Done.

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):
+  filename = '%s/LOG-%s-%s-%03d-%s' % (
+      output_path, window.max_timestamp(), shard, pane.index,
+      pane.timing) if output_path else None
+  return OutputFile(
+      window.max_timestamp(), shard, pane.index, pane.timing, filename)
+
+
+def index_path_for(window):
+  if output_path:
+    return '%s/INDEX-%s' % (output_path, window.max_timestamp())
+  else:
+    return None
+
+
+def load(events, pipeline_options, metadata=None):
+  return (
+      events
+      | 'query10_shard_events' >> beam.ParDo(ShardEventsDoFn())
+      | 'query10_fix_window' >> beam.WindowInto(
+          window.FixedWindows(metadata.get('window_size_sec')),
+          trigger=trigger.AfterEach(

Review comment:
       Done.

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and

Review comment:
       Done

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):

Review comment:
       Done.

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):
+  filename = '%s/LOG-%s-%s-%03d-%s' % (
+      output_path, window.max_timestamp(), shard, pane.index,
+      pane.timing) if output_path else None
+  return OutputFile(
+      window.max_timestamp(), shard, pane.index, pane.timing, filename)
+
+
+def index_path_for(window):
+  if output_path:
+    return '%s/INDEX-%s' % (output_path, window.max_timestamp())
+  else:
+    return None
+
+
+def load(events, pipeline_options, metadata=None):
+  return (
+      events
+      | 'query10_shard_events' >> beam.ParDo(ShardEventsDoFn())
+      | 'query10_fix_window' >> beam.WindowInto(
+          window.FixedWindows(metadata.get('window_size_sec')),
+          trigger=trigger.AfterEach(

Review comment:
       Done.

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689995629


   Run PythonLint PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12807      +/-   ##
   ==========================================
   - Coverage   40.38%   40.34%   -0.04%     
   ==========================================
     Files         451      452       +1     
     Lines       53721    53774      +53     
   ==========================================
     Hits        21696    21696              
   - Misses      32025    32078      +53     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...64aaa82](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on a change in pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12807:
URL: https://github.com/apache/beam/pull/12807#discussion_r486742790



##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12807      +/-   ##
   ==========================================
   - Coverage   40.38%   40.34%   -0.04%     
   ==========================================
     Files         451      452       +1     
     Lines       53721    53774      +53     
   ==========================================
     Hits        21696    21696              
   - Misses      32025    32078      +53     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...64aaa82](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz removed a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz removed a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689995629


   Run PythonLint PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] leiyiz commented on a change in pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
leiyiz commented on a change in pull request #12807:
URL: https://github.com/apache/beam/pull/12807#discussion_r486742739



##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):
+  filename = '%s/LOG-%s-%s-%03d-%s' % (
+      output_path, window.max_timestamp(), shard, pane.index,
+      pane.timing) if output_path else None
+  return OutputFile(
+      window.max_timestamp(), shard, pane.index, pane.timing, filename)
+
+
+def index_path_for(window):
+  if output_path:
+    return '%s/INDEX-%s' % (output_path, window.max_timestamp())
+  else:
+    return None
+
+
+def load(events, pipeline_options, metadata=None):
+  return (
+      events
+      | 'query10_shard_events' >> beam.ParDo(ShardEventsDoFn())
+      | 'query10_fix_window' >> beam.WindowInto(
+          window.FixedWindows(metadata.get('window_size_sec')),
+          trigger=trigger.AfterEach(

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] y1chi commented on a change in pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #12807:
URL: https://github.com/apache/beam/pull/12807#discussion_r486491662



##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):

Review comment:
       nit: add some documentation on what the function is used for.

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and

Review comment:
       Could you link or open a jira issue regarding this and comment it here?

##########
File path: sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query10.py
##########
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Query 10, 'Log to sharded files' (Not in original suite.)
+
+Every window_size_sec, save all events from the last period into
+2*max_workers log files.
+"""
+
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
+from apache_beam.utils.timestamp import Duration
+
+NUM_SHARD_PER_WORKER = 5
+LATE_BATCHING_PERIOD = Duration.of(10)
+
+output_path = None
+max_num_workers = 5
+
+num_log_shards = NUM_SHARD_PER_WORKER * max_num_workers
+
+
+class OutputFile(object):
+  def __init__(self, max_timestamp, shard, index, timing, filename):
+    self.max_timestamp = max_timestamp
+    self.shard = shard
+    self.index = index
+    self.timing = timing
+    self.filename = filename
+
+
+def open_writable_gcs_file(options, filename):
+  # TODO: it seems that beam team has not yet decided about this method and
+  #   it is left blank and unspecified.
+  pass
+
+
+def output_file_for(window, shard, pane):
+  filename = '%s/LOG-%s-%s-%03d-%s' % (
+      output_path, window.max_timestamp(), shard, pane.index,
+      pane.timing) if output_path else None
+  return OutputFile(
+      window.max_timestamp(), shard, pane.index, pane.timing, filename)
+
+
+def index_path_for(window):
+  if output_path:
+    return '%s/INDEX-%s' % (output_path, window.max_timestamp())
+  else:
+    return None
+
+
+def load(events, pipeline_options, metadata=None):
+  return (
+      events
+      | 'query10_shard_events' >> beam.ParDo(ShardEventsDoFn())
+      | 'query10_fix_window' >> beam.WindowInto(
+          window.FixedWindows(metadata.get('window_size_sec')),
+          trigger=trigger.AfterEach(

Review comment:
       Can we add a comment on the desired behavior of this trigger as well?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `41.91%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12807       +/-   ##
   ===========================================
   + Coverage   40.38%   82.30%   +41.91%     
   ===========================================
     Files         451      452        +1     
     Lines       53721    53803       +82     
   ===========================================
   + Hits        21696    44280    +22584     
   + Misses      32025     9523    -22502     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | [...runners/interactive/options/interactive\_options.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2ludGVyYWN0aXZlX29wdGlvbnMucHk=) | `86.66% <0.00%> (+6.66%)` | :arrow_up: |
   | ... and [270 more](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...0c716e2](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12807: [BEAM-2855] implement query 10

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12807:
URL: https://github.com/apache/beam/pull/12807#issuecomment-689997960


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=h1) Report
   > Merging [#12807](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12807/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12807      +/-   ##
   ==========================================
   - Coverage   40.38%   40.34%   -0.04%     
   ==========================================
     Files         451      452       +1     
     Lines       53721    53774      +53     
   ==========================================
     Hits        21696    21696              
   - Misses      32025    32078      +53     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12807/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=footer). Last update [5705526...64aaa82](https://codecov.io/gh/apache/beam/pull/12807?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org