You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/16 02:38:24 UTC
[GitHub] [beam] dennisylyung opened a new pull request #14244: [BEAM-11907] SQS source with full functionality
dennisylyung opened a new pull request #14244:
URL: https://github.com/apache/beam/pull/14244
The current SqsIO source connector has limited functionality.
This PR implements the following functionalities to SqsIO:
- Batch message deletion (mitigate throttled deleteMessage requests)
- Extension of visibility timeout (The existing implementation causes message duplication)
- Update watermarks when no new messages
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_Post
Commit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ld/srcs/sdks/python/apache\_beam/utils/processes.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvY2Vzc2VzLnB5) | | |
| [.../snippets/transforms/elementwise/withtimestamps.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS93aXRodGltZXN0YW1wcy5weQ==) | | |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | | |
| [...apache\_beam/examples/complete/game/leader\_board.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9sZWFkZXJfYm9hcmQucHk=) | | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | | |
| [...hon/apache\_beam/io/gcp/datastore/v1new/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9fX2luaXRfXy5weQ==) | | |
| [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | | |
| [...ers/interactive/display/pipeline\_graph\_renderer.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3BpcGVsaW5lX2dyYXBoX3JlbmRlcmVyLnB5) | | |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | | |
| [...e\_beam/examples/complete/juliaset/juliaset\_main.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXRfbWFpbi5weQ==) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head d0463a7. Consider uploading reports for the commit d0463a7 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...\_beam/testing/benchmarks/nexmark/queries/query2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTIucHk=) | | |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/utils/counters.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvY291bnRlcnMucHk=) | | |
| [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | | |
| [...build/srcs/sdks/python/apache\_beam/utils/plugin.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcGx1Z2luLnB5) | | |
| [...uild/srcs/sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | | |
| [...on/apache\_beam/runners/worker/statesampler\_slow.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc3RhdGVzYW1wbGVyX3Nsb3cucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query5.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTUucHk=) | | |
| [...s/python/apache\_beam/examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | | |
| [...s/sdks/python/apache\_beam/examples/avro\_bitcoin.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvYXZyb19iaXRjb2luLnB5) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...d0463a7](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r634003803
##########
File path: sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedReader.java
##########
@@ -17,51 +17,385 @@
*/
package org.apache.beam.sdk.io.aws.sqs;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.toMap;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.amazonaws.services.sqs.AmazonSQS;
+import com.amazonaws.services.sqs.model.BatchResultErrorEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchRequestEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchResult;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchRequestEntry;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchResult;
+import com.amazonaws.services.sqs.model.GetQueueAttributesRequest;
import com.amazonaws.services.sqs.model.Message;
+import com.amazonaws.services.sqs.model.MessageAttributeValue;
import com.amazonaws.services.sqs.model.MessageSystemAttributeName;
+import com.amazonaws.services.sqs.model.QueueAttributeName;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageResult;
-import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.io.IOException;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
+import java.util.Objects;
import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.BucketingFunction;
+import org.apache.beam.sdk.util.MovingFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.EvictingQueue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
@SuppressWarnings({
"nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
})
-class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> implements Serializable {
+class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> {
+ private static final Logger LOG = LoggerFactory.getLogger(SqsUnboundedReader.class);
+ /** Maximum number of messages to pull from SQS per request. */
public static final int MAX_NUMBER_OF_MESSAGES = 10;
+
+ /** Maximum times to retry batch SQS operations upon partial success. */
+ private static final int BATCH_OPERATION_MAX_RETIRES = 5;
Review comment:
To be honest most are copied from the pubsub connector, except the update and delete batch size of 10, which is the limit of the AWS API.
Please advise if there are better ways to determine the constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r629317202
##########
File path: sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedReader.java
##########
@@ -17,51 +17,385 @@
*/
package org.apache.beam.sdk.io.aws.sqs;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.toMap;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.amazonaws.services.sqs.AmazonSQS;
+import com.amazonaws.services.sqs.model.BatchResultErrorEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchRequestEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchResult;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchRequestEntry;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchResult;
+import com.amazonaws.services.sqs.model.GetQueueAttributesRequest;
import com.amazonaws.services.sqs.model.Message;
+import com.amazonaws.services.sqs.model.MessageAttributeValue;
import com.amazonaws.services.sqs.model.MessageSystemAttributeName;
+import com.amazonaws.services.sqs.model.QueueAttributeName;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageResult;
-import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.io.IOException;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
+import java.util.Objects;
import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.BucketingFunction;
+import org.apache.beam.sdk.util.MovingFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.EvictingQueue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
@SuppressWarnings({
"nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
})
-class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> implements Serializable {
+class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> {
+ private static final Logger LOG = LoggerFactory.getLogger(SqsUnboundedReader.class);
+ /** Maximum number of messages to pull from SQS per request. */
public static final int MAX_NUMBER_OF_MESSAGES = 10;
+
+ /** Maximum times to retry batch SQS operations upon partial success. */
+ private static final int BATCH_OPERATION_MAX_RETIRES = 5;
Review comment:
How the values of this and other constants below were estimated?
##########
File path: sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java
##########
@@ -46,25 +52,151 @@
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
@Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
- @Test
- public void testRead() {
+ private SqsUnboundedSource source;
+
+ private void setupOneMessage() {
final AmazonSQS client = embeddedSqsRestServer.getClient();
final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(queueUrl, DATA);
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final AmazonSQS client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(queueUrl, message);
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ @Test
+ public void checkpointCoderIsSane() {
Review comment:
Please, add "test" prefix for all test methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 87fb70c. Consider uploading reports for the commit 87fb70c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/display/pipeline\_graph.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3BpcGVsaW5lX2dyYXBoLnB5) | | |
| [...python/apache\_beam/runners/portability/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9fX2luaXRfXy5weQ==) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTMucHk=) | | |
| [...hon/apache\_beam/examples/wordcount\_with\_metrics.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X3dpdGhfbWV0cmljcy5weQ==) | | |
| [...gcp/datastore/v1new/datastore\_write\_it\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVfd3JpdGVfaXRfcGlwZWxpbmUucHk=) | | |
| [...srcs/sdks/python/apache\_beam/io/gcp/dicomclient.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RpY29tY2xpZW50LnB5) | | |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | | |
| [...py38/build/srcs/sdks/python/apache\_beam/version.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdmVyc2lvbi5weQ==) | | |
| [...rcs/sdks/python/apache\_beam/transforms/timeutil.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90aW1ldXRpbC5weQ==) | | |
| [...ache\_beam/testing/benchmarks/chicago\_taxi/setup.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9zZXR1cC5weQ==) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...87fb70c](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-872563313
> If there are no principal objections then I think it can be merged.
I agree. There is one approval, and all tests pass. Feel free to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-847266933
@dpcollins-google / @boyuanzz - Would one of you also review this change? (@aromanenko-dev mentioned that most of the change is based from the pubsub io.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 71711ec. Consider uploading reports for the commit 71711ec to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...dks/python/apache\_beam/portability/api/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL19faW5pdF9fLnB5) | | |
| [...srcs/sdks/python/apache\_beam/io/gcp/tests/utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL3V0aWxzLnB5) | | |
| [...m/portability/api/beam\_interactive\_api\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1faW50ZXJhY3RpdmVfYXBpX3BiMl9ncnBjLnB5) | | |
| [...ache\_beam/coders/proto2\_coder\_test\_messages\_pb2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Byb3RvMl9jb2Rlcl90ZXN0X21lc3NhZ2VzX3BiMi5weQ==) | | |
| [...cs/sdks/python/apache\_beam/portability/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvX19pbml0X18ucHk=) | | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | | |
| [...es/snippets/transforms/aggregation/cogroupbykey.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb2dyb3VwYnlrZXkucHk=) | | |
| [...cp/internal/clients/storage/storage\_v1\_messages.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvc3RvcmFnZS9zdG9yYWdlX3YxX21lc3NhZ2VzLnB5) | | |
| [...\_beam/io/gcp/internal/clients/bigquery/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvYmlncXVlcnkvX19pbml0X18ucHk=) | | |
| [...n/apache\_beam/runners/direct/evaluation\_context.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXZhbHVhdGlvbl9jb250ZXh0LnB5) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...71711ec](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.79%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head c3fa627. Consider uploading reports for the commit c3fa627 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.65% 82.86% -0.80%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49558 47720 -1838
- Misses 9681 9867 +186
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...es/snippets/transforms/aggregation/cogroupbykey.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb2dyb3VwYnlrZXkucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/internal/gcp/auth.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZ2NwL2F1dGgucHk=) | | |
| [.../sdks/python/apache\_beam/examples/snippets/util.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdXRpbC5weQ==) | | |
| [...ython/apache\_beam/io/gcp/bigquery\_read\_internal.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3JlYWRfaW50ZXJuYWwucHk=) | | |
| [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | | |
| [...ache\_beam/coders/proto2\_coder\_test\_messages\_pb2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Byb3RvMl9jb2Rlcl90ZXN0X21lc3NhZ2VzX3BiMi5weQ==) | | |
| [...ython/apache\_beam/io/aws/clients/s3/fake\_client.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvZmFrZV9jbGllbnQucHk=) | | |
| [...m/portability/api/beam\_interactive\_api\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1faW50ZXJhY3RpdmVfYXBpX3BiMl9ncnBjLnB5) | | |
| [...s/sdks/python/apache\_beam/runners/worker/logger.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvbG9nZ2VyLnB5) | | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...c3fa627](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r634003803
##########
File path: sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedReader.java
##########
@@ -17,51 +17,385 @@
*/
package org.apache.beam.sdk.io.aws.sqs;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.toMap;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.amazonaws.services.sqs.AmazonSQS;
+import com.amazonaws.services.sqs.model.BatchResultErrorEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchRequestEntry;
+import com.amazonaws.services.sqs.model.ChangeMessageVisibilityBatchResult;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchRequestEntry;
+import com.amazonaws.services.sqs.model.DeleteMessageBatchResult;
+import com.amazonaws.services.sqs.model.GetQueueAttributesRequest;
import com.amazonaws.services.sqs.model.Message;
+import com.amazonaws.services.sqs.model.MessageAttributeValue;
import com.amazonaws.services.sqs.model.MessageSystemAttributeName;
+import com.amazonaws.services.sqs.model.QueueAttributeName;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageResult;
-import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.io.IOException;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
+import java.util.Objects;
import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.BucketingFunction;
+import org.apache.beam.sdk.util.MovingFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.EvictingQueue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
@SuppressWarnings({
"nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
})
-class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> implements Serializable {
+class SqsUnboundedReader extends UnboundedSource.UnboundedReader<Message> {
+ private static final Logger LOG = LoggerFactory.getLogger(SqsUnboundedReader.class);
+ /** Maximum number of messages to pull from SQS per request. */
public static final int MAX_NUMBER_OF_MESSAGES = 10;
+
+ /** Maximum times to retry batch SQS operations upon partial success. */
+ private static final int BATCH_OPERATION_MAX_RETIRES = 5;
Review comment:
To be honest most are copied from the pubsub connector, except the update and delete batch size of 10, which is the limit of the AWS API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...m/runners/portability/flink\_uber\_jar\_job\_server.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbGlua191YmVyX2phcl9qb2Jfc2VydmVyLnB5) | | |
| [...uild/srcs/sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/io/gcp/dicomclient.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RpY29tY2xpZW50LnB5) | | |
| [...examples/snippets/transforms/aggregation/latest.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9sYXRlc3QucHk=) | | |
| [...thon/apache\_beam/runners/portability/job\_server.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9qb2Jfc2VydmVyLnB5) | | |
| [...ython/apache\_beam/io/aws/clients/s3/fake\_client.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvZmFrZV9jbGllbnQucHk=) | | |
| [...thon/apache\_beam/testing/metric\_result\_matchers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9tZXRyaWNfcmVzdWx0X21hdGNoZXJzLnB5) | | |
| [...hon/apache\_beam/portability/api/schema\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3NjaGVtYV9wYjJfZ3JwYy5weQ==) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-847267732
@boyuanzz is probably a better reviewer, but I can if they're too busy. Reach out to me internally if you need my review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-822929832
@dennisylyung - Thank you!
Not sure who might be a good reviewer for this PR. Do you mind pinging the dev@ thread you started (or a new thread) ask for reviews?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 03d3af9. Consider uploading reports for the commit 03d3af9 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | | |
| [...ython/apache\_beam/examples/kafkataxi/kafka\_taxi.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMva2Fma2F0YXhpL2thZmthX3RheGkucHk=) | | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | | |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | | |
| [...ld/srcs/sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | | |
| [...am/portability/api/standard\_window\_fns\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3N0YW5kYXJkX3dpbmRvd19mbnNfcGIyX2dycGMucHk=) | | |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | | |
| [...cs/sdks/python/apache\_beam/dataframe/transforms.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3RyYW5zZm9ybXMucHk=) | | |
| [...rcs/sdks/python/apache\_beam/typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...03d3af9](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r654247286
##########
File path: sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java
##########
@@ -46,25 +52,151 @@
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
@Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
- @Test
- public void testRead() {
+ private SqsUnboundedSource source;
+
+ private void setupOneMessage() {
final AmazonSQS client = embeddedSqsRestServer.getClient();
final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(queueUrl, DATA);
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final AmazonSQS client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(queueUrl, message);
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ @Test
+ public void testCheckpointCoderIsSane() {
+ setupOneMessage();
+ CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
+ // Since we only serialize/deserialize the 'notYetReadIds', and we don't want to make
+ // equals on checkpoints ignore those fields, we'll test serialization and deserialization
+ // of checkpoints in multipleReaders below.
+ }
- final PCollection<Message> output =
- pipeline.apply(SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(100));
+ @Test
+ public void testReadOneMessage() throws IOException {
Review comment:
changed in the new commit
##########
File path: sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/SqsIOTest.java
##########
@@ -17,59 +17,200 @@
*/
package org.apache.beam.sdk.io.aws2.sqs;
+import static junit.framework.TestCase.assertFalse;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
-import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
+import org.apache.beam.sdk.util.CoderUtils;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import software.amazon.awssdk.services.sqs.SqsClient;
+import software.amazon.awssdk.services.sqs.model.ChangeMessageVisibilityRequest;
import software.amazon.awssdk.services.sqs.model.Message;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse;
import software.amazon.awssdk.services.sqs.model.SendMessageRequest;
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
+@SuppressWarnings({
+ "nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
+})
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
- @Test
- public void testRead() {
- final SqsClient client = EmbeddedSqsServer.getClient();
- final String queueUrl = EmbeddedSqsServer.getQueueUrl();
+ @Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
+
+ private SqsUnboundedSource source;
- final PCollection<SqsMessage> output =
- pipeline.apply(
+ private void setupOneMessage() {
+ final SqsClient client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(SendMessageRequest.builder().queueUrl(queueUrl).messageBody(DATA).build());
+ source =
+ new SqsUnboundedSource(
SqsIO.read()
+ .withQueueUrl(queueUrl)
.withSqsClientProvider(SqsClientProviderMock.of(client))
+ .withMaxNumRecords(1));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final SqsClient client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(
+ SendMessageRequest.builder().queueUrl(queueUrl).messageBody(message).build());
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read()
.withQueueUrl(queueUrl)
- .withMaxNumRecords(100));
+ .withSqsClientProvider(SqsClientProviderMock.of(client))
+ .withMaxNumRecords(messages.size()));
+ }
- PAssert.thatSingleton(output.apply(Count.globally())).isEqualTo(100L);
+ @Test
+ public void testCheckpointCoderIsSane() {
+ setupOneMessage();
+ CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
+ // Since we only serialize/deserialize the 'notYetReadIds', and we don't want to make
+ // equals on checkpoints ignore those fields, we'll test serialization and deserialization
+ // of checkpoints in multipleReaders below.
+ }
+ @Test
+ public void testReadOneMessage() throws IOException {
Review comment:
changed in the new commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head ada287a. Consider uploading reports for the commit ada287a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../build/srcs/sdks/python/apache\_beam/io/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vX19pbml0X18ucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query4.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTQucHk=) | | |
| [...thon/apache\_beam/examples/complete/autocomplete.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvYXV0b2NvbXBsZXRlLnB5) | | |
| [...nners/portability/fn\_api\_runner/trigger\_manager.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3RyaWdnZXJfbWFuYWdlci5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/transforms/display.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9kaXNwbGF5LnB5) | | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | | |
| [...hon/apache\_beam/examples/wordcount\_with\_metrics.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X3dpdGhfbWV0cmljcy5weQ==) | | |
| [...ython/apache\_beam/portability/api/endpoints\_pb2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2VuZHBvaW50c19wYjIucHk=) | | |
| [...sdks/python/apache\_beam/utils/subprocess\_server.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvc3VicHJvY2Vzc19zZXJ2ZXIucHk=) | | |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...ada287a](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #14244:
URL: https://github.com/apache/beam/pull/14244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-825732792
@dennisylyung Thanks! I'll take a look in the next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/a834a7534a48f2be3f5e1510630526105335d7dd?el=desc) (a834a75) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head b15a02c. Consider uploading reports for the commit b15a02c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.05% 82.86% -0.19%
==========================================
Files 469 466 -3
Lines 58472 57587 -885
==========================================
- Hits 48563 47720 -843
+ Misses 9909 9867 -42
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...cs/sdks/python/apache\_beam/portability/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvX19pbml0X18ucHk=) | | |
| [...beam/runners/portability/local\_job\_service\_main.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZV9tYWluLnB5) | | |
| [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | | |
| [...hon/apache\_beam/portability/api/schema\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3NjaGVtYV9wYjJfZ3JwYy5weQ==) | | |
| [...38/build/srcs/sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | | |
| [.../srcs/sdks/python/apache\_beam/io/range\_trackers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcmFuZ2VfdHJhY2tlcnMucHk=) | | |
| [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | | |
| [...ks/python/apache\_beam/examples/cookbook/filters.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svZmlsdGVycy5weQ==) | | |
| [...pache\_beam/runners/interactive/caching/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL19faW5pdF9fLnB5) | | |
| [...he\_beam/examples/cookbook/multiple\_output\_pardo.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svbXVsdGlwbGVfb3V0cHV0X3BhcmRvLnB5) | | |
| ... and [925 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [a834a75...b15a02c](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-824597552
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 231518f. Consider uploading reports for the commit 231518f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | | |
| [.../sdks/python/apache\_beam/examples/snippets/util.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdXRpbC5weQ==) | | |
| [...examples/snippets/transforms/elementwise/filter.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9maWx0ZXIucHk=) | | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | | |
| [...ache\_beam/testing/benchmarks/chicago\_taxi/setup.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9zZXR1cC5weQ==) | | |
| [...d/srcs/sdks/python/apache\_beam/io/filebasedsink.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc2luay5weQ==) | | |
| [...38/build/srcs/sdks/python/apache\_beam/io/textio.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vdGV4dGlvLnB5) | | |
| [.../srcs/sdks/python/apache\_beam/metrics/execution.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9leGVjdXRpb24ucHk=) | | |
| [.../sdks/python/apache\_beam/runners/internal/names.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcm5hbC9uYW1lcy5weQ==) | | |
| [...sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...231518f](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.79%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 6664a8d. Consider uploading reports for the commit 6664a8d to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.65% 82.86% -0.80%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49558 47720 -1838
- Misses 9681 9867 +186
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...nchmarks/chicago\_taxi/tfdv\_analyze\_and\_validate.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS90ZmR2X2FuYWx5emVfYW5kX3ZhbGlkYXRlLnB5) | | |
| [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | | |
| [...d/srcs/sdks/python/apache\_beam/options/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9fX2luaXRfXy5weQ==) | | |
| [...hon/apache\_beam/examples/cookbook/mergecontacts.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svbWVyZ2Vjb250YWN0cy5weQ==) | | |
| [.../runners/interactive/testing/pipeline\_assertion.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL3BpcGVsaW5lX2Fzc2VydGlvbi5weQ==) | | |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | | |
| [.../gcp/internal/clients/storage/storage\_v1\_client.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvc3RvcmFnZS9zdG9yYWdlX3YxX2NsaWVudC5weQ==) | | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | | |
| [...uild/srcs/sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | | |
| [.../srcs/sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...6664a8d](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-800463113
cc: @boyuanzz
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/a834a7534a48f2be3f5e1510630526105335d7dd?el=desc) (a834a75) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 3e8160b. Consider uploading reports for the commit 3e8160b to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.05% 82.86% -0.19%
==========================================
Files 469 466 -3
Lines 58472 57587 -885
==========================================
- Hits 48563 47720 -843
+ Misses 9909 9867 -42
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...srcs/sdks/python/apache\_beam/dataframe/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL19faW5pdF9fLnB5) | | |
| [...rcs/sdks/python/apache\_beam/transforms/timeutil.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90aW1ldXRpbC5weQ==) | | |
| [...ld/srcs/sdks/python/apache\_beam/coders/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL19faW5pdF9fLnB5) | | |
| [...ython/apache\_beam/runners/direct/bundle\_factory.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvYnVuZGxlX2ZhY3RvcnkucHk=) | | |
| [...on/apache\_beam/runners/direct/watermark\_manager.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvd2F0ZXJtYXJrX21hbmFnZXIucHk=) | | |
| [...sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | | |
| [...m/runners/portability/flink\_uber\_jar\_job\_server.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbGlua191YmVyX2phcl9qb2Jfc2VydmVyLnB5) | | |
| [.../apache\_beam/ml/gcp/videointelligenceml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3ZpZGVvaW50ZWxsaWdlbmNlbWxfdGVzdF9pdC5weQ==) | | |
| [...python/apache\_beam/transforms/external\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbF90ZXN0X3B5My5weQ==) | | |
| [.../sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | | |
| ... and [925 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [a834a75...3e8160b](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-824193580
@iemejia Sure, I'll take a look as soon as I can (I hope in a couple of next days).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...he\_beam/transforms/combinefn\_lifecycle\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lZm5fbGlmZWN5Y2xlX3BpcGVsaW5lLnB5) | | |
| [...ld/srcs/sdks/python/apache\_beam/io/filesystemio.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbWlvLnB5) | | |
| [...on/apache\_beam/runners/direct/helper\_transforms.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvaGVscGVyX3RyYW5zZm9ybXMucHk=) | | |
| [...pache\_beam/runners/portability/artifact\_service.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hcnRpZmFjdF9zZXJ2aWNlLnB5) | | |
| [...cs/sdks/python/apache\_beam/transforms/userstate.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91c2Vyc3RhdGUucHk=) | | |
| [...am/examples/snippets/transforms/aggregation/min.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9taW4ucHk=) | | |
| [...cp/internal/clients/storage/storage\_v1\_messages.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvc3RvcmFnZS9zdG9yYWdlX3YxX21lc3NhZ2VzLnB5) | | |
| [...cs/sdks/python/apache\_beam/utils/windowed\_value.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvd2luZG93ZWRfdmFsdWUucHk=) | | |
| [...ers/dataflow/dataflow\_exercise\_metrics\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19leGVyY2lzZV9tZXRyaWNzX3BpcGVsaW5lLnB5) | | |
| [...e\_beam/examples/complete/game/hourly\_team\_score.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9ob3VybHlfdGVhbV9zY29yZS5weQ==) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-833927832
> @dennisylyung Thanks! I'll take a look in the next week.
@aromanenko-dev, thank you and a gentle ping to get this in your review queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-816334092
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-863761156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-822947208
@iemejia Do you think you can review, or ping someone who is suitable? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-825393917
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r637911365
##########
File path: sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/SqsIOTest.java
##########
@@ -17,59 +17,200 @@
*/
package org.apache.beam.sdk.io.aws2.sqs;
+import static junit.framework.TestCase.assertFalse;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
-import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
+import org.apache.beam.sdk.util.CoderUtils;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import software.amazon.awssdk.services.sqs.SqsClient;
+import software.amazon.awssdk.services.sqs.model.ChangeMessageVisibilityRequest;
import software.amazon.awssdk.services.sqs.model.Message;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse;
import software.amazon.awssdk.services.sqs.model.SendMessageRequest;
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
+@SuppressWarnings({
+ "nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
+})
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
- @Test
- public void testRead() {
- final SqsClient client = EmbeddedSqsServer.getClient();
- final String queueUrl = EmbeddedSqsServer.getQueueUrl();
+ @Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
+
+ private SqsUnboundedSource source;
- final PCollection<SqsMessage> output =
- pipeline.apply(
+ private void setupOneMessage() {
+ final SqsClient client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(SendMessageRequest.builder().queueUrl(queueUrl).messageBody(DATA).build());
+ source =
+ new SqsUnboundedSource(
SqsIO.read()
+ .withQueueUrl(queueUrl)
.withSqsClientProvider(SqsClientProviderMock.of(client))
+ .withMaxNumRecords(1));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final SqsClient client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(
+ SendMessageRequest.builder().queueUrl(queueUrl).messageBody(message).build());
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read()
.withQueueUrl(queueUrl)
- .withMaxNumRecords(100));
+ .withSqsClientProvider(SqsClientProviderMock.of(client))
+ .withMaxNumRecords(messages.size()));
+ }
- PAssert.thatSingleton(output.apply(Count.globally())).isEqualTo(100L);
+ @Test
+ public void testCheckpointCoderIsSane() {
+ setupOneMessage();
+ CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
+ // Since we only serialize/deserialize the 'notYetReadIds', and we don't want to make
+ // equals on checkpoints ignore those fields, we'll test serialization and deserialization
+ // of checkpoints in multipleReaders below.
+ }
+ @Test
+ public void testReadOneMessage() throws IOException {
Review comment:
Please, move all `Reader`'s tests to a dedicated test class. e.g. `SqsUnboundedReaderTest`
##########
File path: sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java
##########
@@ -46,25 +52,151 @@
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
@Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
- @Test
- public void testRead() {
+ private SqsUnboundedSource source;
+
+ private void setupOneMessage() {
final AmazonSQS client = embeddedSqsRestServer.getClient();
final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(queueUrl, DATA);
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final AmazonSQS client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(queueUrl, message);
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ @Test
+ public void testCheckpointCoderIsSane() {
+ setupOneMessage();
+ CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
+ // Since we only serialize/deserialize the 'notYetReadIds', and we don't want to make
+ // equals on checkpoints ignore those fields, we'll test serialization and deserialization
+ // of checkpoints in multipleReaders below.
+ }
- final PCollection<Message> output =
- pipeline.apply(SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(100));
+ @Test
+ public void testReadOneMessage() throws IOException {
Review comment:
Please, move all `Reader`'s tests to a dedicated test class. e.g. `SqsUnboundedReaderTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | | |
| [...che\_beam/portability/api/beam\_provision\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcHJvdmlzaW9uX2FwaV9wYjIucHk=) | | |
| [...amples/snippets/transforms/aggregation/distinct.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9kaXN0aW5jdC5weQ==) | | |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | | |
| [...on/apache\_beam/io/gcp/internal/clients/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvX19pbml0X18ucHk=) | | |
| [.../internal/clients/bigquery/bigquery\_v2\_messages.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvYmlncXVlcnkvYmlncXVlcnlfdjJfbWVzc2FnZXMucHk=) | | |
| [...eam/portability/api/beam\_expansion\_api\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fZXhwYW5zaW9uX2FwaV9wYjJfZ3JwYy5weQ==) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query1.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query4.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTQucHk=) | | |
| [.../snippets/transforms/elementwise/withtimestamps.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS93aXRodGltZXN0YW1wcy5weQ==) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-817430793
R: @aaltay
It is ready for review now, thanks!
The PR turned out to be more rewriting the connectors referencing the pubsub one than doing small enhancements, since when I was developing a job with the sqs reader I found that it lacks many functionalities to make it generally usable.
Please let me know how to properly document such changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/9ab200ca616b131c31d8a7955d3a380b2e1feed7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9ab200c) will **decrease** coverage by `0.81%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 3948f03. Consider uploading reports for the commit 3948f03 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.68% 82.86% -0.82%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49574 47720 -1854
- Misses 9665 9867 +202
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/examples/wordcount.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50LnB5) | | |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | | |
| [...python/apache\_beam/transforms/external\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbF90ZXN0X3B5My5weQ==) | | |
| [...\_beam/io/gcp/datastore/v1new/adaptive\_throttler.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9hZGFwdGl2ZV90aHJvdHRsZXIucHk=) | | |
| [...cs/sdks/python/apache\_beam/transforms/resources.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9yZXNvdXJjZXMucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query4.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTQucHk=) | | |
| [...thon/apache\_beam/testing/metric\_result\_matchers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9tZXRyaWNfcmVzdWx0X21hdGNoZXJzLnB5) | | |
| [...examples/snippets/transforms/elementwise/values.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS92YWx1ZXMucHk=) | | |
| [...che\_beam/runners/interactive/caching/read\_cache.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlYWRfY2FjaGUucHk=) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...3948f03](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on a change in pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on a change in pull request #14244:
URL: https://github.com/apache/beam/pull/14244#discussion_r634003498
##########
File path: sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java
##########
@@ -46,25 +52,151 @@
/** Tests on {@link SqsIO}. */
@RunWith(JUnit4.class)
public class SqsIOTest {
+ private static final String DATA = "testData";
@Rule public TestPipeline pipeline = TestPipeline.create();
@Rule public EmbeddedSqsServer embeddedSqsRestServer = new EmbeddedSqsServer();
- @Test
- public void testRead() {
+ private SqsUnboundedSource source;
+
+ private void setupOneMessage() {
final AmazonSQS client = embeddedSqsRestServer.getClient();
final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ client.sendMessage(queueUrl, DATA);
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ private void setupMessages(List<String> messages) {
+ final AmazonSQS client = embeddedSqsRestServer.getClient();
+ final String queueUrl = embeddedSqsRestServer.getQueueUrl();
+ for (String message : messages) {
+ client.sendMessage(queueUrl, message);
+ }
+ source =
+ new SqsUnboundedSource(
+ SqsIO.read().withQueueUrl(queueUrl).withMaxNumRecords(1),
+ new SqsConfiguration(pipeline.getOptions().as(AwsOptions.class)));
+ }
+
+ @Test
+ public void checkpointCoderIsSane() {
Review comment:
changed in the new commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 52d7400. Consider uploading reports for the commit 52d7400 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...\_beam/io/gcp/internal/clients/bigquery/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvYmlncXVlcnkvX19pbml0X18ucHk=) | | |
| [...thon/apache\_beam/runners/worker/operation\_specs.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvb3BlcmF0aW9uX3NwZWNzLnB5) | | |
| [.../srcs/sdks/python/apache\_beam/io/range\_trackers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcmFuZ2VfdHJhY2tlcnMucHk=) | | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | | |
| [...examples/snippets/transforms/elementwise/kvswap.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9rdnN3YXAucHk=) | | |
| [...pache\_beam/runners/portability/artifact\_service.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hcnRpZmFjdF9zZXJ2aWNlLnB5) | | |
| [...thon/apache\_beam/examples/complete/autocomplete.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvYXV0b2NvbXBsZXRlLnB5) | | |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | | |
| [...s/python/apache\_beam/testing/pipeline\_verifiers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9waXBlbGluZV92ZXJpZmllcnMucHk=) | | |
| [...on/apache\_beam/runners/worker/statesampler\_slow.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc3RhdGVzYW1wbGVyX3Nsb3cucHk=) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...52d7400](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-836629862
Sorry, I was quite busy last week. I'll try to take a look asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-847266933
@dpcollins-google / @boyuanzz - Would one of you also review this change? (@aromanenko-dev mentioned that most of the change is based on the pubsub io implementation.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-823166296
@dennisylyung I am a bit busy this week but maybe @aromanenko-dev can take a look. In the main time can you please rebase and squash your commits. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-823751195
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...cs/sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | | |
| [...dks/python/apache\_beam/io/gcp/internal/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL19faW5pdF9fLnB5) | | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | | |
| [...am/examples/snippets/transforms/elementwise/map.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9tYXAucHk=) | | |
| [...beam/testing/benchmarks/chicago\_taxi/preprocess.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9wcmVwcm9jZXNzLnB5) | | |
| [...ild/srcs/sdks/python/apache\_beam/utils/counters.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvY291bnRlcnMucHk=) | | |
| [.../internal/clients/bigquery/bigquery\_v2\_messages.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvYmlncXVlcnkvYmlncXVlcnlfdjJfbWVzc2FnZXMucHk=) | | |
| [...uild/srcs/sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | | |
| [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/a834a7534a48f2be3f5e1510630526105335d7dd?el=desc) (a834a75) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 2010830. Consider uploading reports for the commit 2010830 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.05% 82.86% -0.19%
==========================================
Files 469 466 -3
Lines 58472 57587 -885
==========================================
- Hits 48563 47720 -843
+ Misses 9909 9867 -42
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...srcs/sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | | |
| [...beam/runners/portability/fn\_api\_runner/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL19faW5pdF9fLnB5) | | |
| [...on/apache\_beam/runners/portability/flink\_runner.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbGlua19ydW5uZXIucHk=) | | |
| [.../build/srcs/sdks/python/apache\_beam/tools/utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvdXRpbHMucHk=) | | |
| [...rcs/sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | | |
| [...sdks/python/apache\_beam/io/restriction\_trackers.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcmVzdHJpY3Rpb25fdHJhY2tlcnMucHk=) | | |
| [...dks/python/apache\_beam/ml/gcp/naturallanguageml.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sLnB5) | | |
| [...ython/apache\_beam/typehints/decorators\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2RlY29yYXRvcnNfdGVzdF9weTMucHk=) | | |
| [...amples/snippets/transforms/aggregation/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9fX2luaXRfXy5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/utils/sentinel.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvc2VudGluZWwucHk=) | | |
| ... and [925 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [a834a75...2010830](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-824195747
In the same time, I see that these 2 gradle tasks failed:
`:sdks:java:io:amazon-web-services:compileJava`
`:sdks:java:io:amazon-web-services2:compileJava`
More details are [here](https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/17309/consoleFull)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head 41b3d95. Consider uploading reports for the commit 41b3d95 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query9.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTkucHk=) | | |
| [...n/apache\_beam/examples/cookbook/bigquery\_schema.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfc2NoZW1hLnB5) | | |
| [...eam/transforms/py\_dataflow\_distribution\_counter.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9weV9kYXRhZmxvd19kaXN0cmlidXRpb25fY291bnRlci5weQ==) | | |
| [...pache\_beam/typehints/trivial\_inference\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3RyaXZpYWxfaW5mZXJlbmNlX3Rlc3RfcHkzLnB5) | | |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | | |
| [...hon/apache\_beam/examples/cookbook/mergecontacts.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svbWVyZ2Vjb250YWN0cy5weQ==) | | |
| [...ks/python/apache\_beam/runners/dataflow/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9fX2luaXRfXy5weQ==) | | |
| [...am/examples/snippets/transforms/aggregation/min.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9taW4ucHk=) | | |
| [...n/apache\_beam/examples/complete/game/game\_stats.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9nYW1lX3N0YXRzLnB5) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...41b3d95](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-863971088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-859195741
@boyuanzz - Would you be able to review this when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-825442817
@aromanenko-dev It passes the java precommit tests now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-823902625
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/56d7e1b0061cd1c0ebfdb022a4e093ae1092f2f3?el=desc) (56d7e1b) will **decrease** coverage by `0.47%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.34% 82.86% -0.48%
==========================================
Files 468 466 -2
Lines 58801 57587 -1214
==========================================
- Hits 49008 47720 -1288
- Misses 9793 9867 +74
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ild/srcs/sdks/python/apache\_beam/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tZXRyaWMucHk=) | | |
| [...srcs/sdks/python/apache\_beam/typehints/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL19faW5pdF9fLnB5) | | |
| [...he\_beam/examples/cookbook/multiple\_output\_pardo.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svbXVsdGlwbGVfb3V0cHV0X3BhcmRvLnB5) | | |
| [...on/apache\_beam/runners/direct/helper\_transforms.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvaGVscGVyX3RyYW5zZm9ybXMucHk=) | | |
| [...thon/apache\_beam/runners/portability/job\_server.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9qb2Jfc2VydmVyLnB5) | | |
| [...e\_beam/examples/complete/game/hourly\_team\_score.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9ob3VybHlfdGVhbV9zY29yZS5weQ==) | | |
| [.../srcs/sdks/python/apache\_beam/portability/utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvdXRpbHMucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query7.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTcucHk=) | | |
| [...sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | | |
| [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | | |
| ... and [924 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [56d7e1b...0f88155](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/a834a7534a48f2be3f5e1510630526105335d7dd?el=desc) (a834a75) will **decrease** coverage by `0.18%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head a31475f. Consider uploading reports for the commit a31475f to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.05% 82.86% -0.19%
==========================================
Files 469 466 -3
Lines 58472 57587 -885
==========================================
- Hits 48563 47720 -843
+ Misses 9909 9867 -42
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query9.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTkucHk=) | | |
| [...ython/apache\_beam/typehints/decorators\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2RlY29yYXRvcnNfdGVzdF9weTMucHk=) | | |
| [...s/sdks/python/apache\_beam/internal/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZ2NwL19faW5pdF9fLnB5) | | |
| [...sdks/python/apache\_beam/examples/flink/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvZmxpbmsvX19pbml0X18ucHk=) | | |
| [...cs/sdks/python/apache\_beam/io/external/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvX19pbml0X18ucHk=) | | |
| [...am/examples/snippets/transforms/elementwise/map.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9tYXAucHk=) | | |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | | |
| [...ow/dataflow\_exercise\_streaming\_metrics\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19leGVyY2lzZV9zdHJlYW1pbmdfbWV0cmljc19waXBlbGluZS5weQ==) | | |
| [...nchmarks/chicago\_taxi/tfdv\_analyze\_and\_validate.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS90ZmR2X2FuYWx5emVfYW5kX3ZhbGlkYXRlLnB5) | | |
| ... and [925 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer). Last update [a834a75...a31475f](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #14244:
URL: https://github.com/apache/beam/pull/14244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-799904516
# [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14244](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c2264f) into [master](https://codecov.io/gh/apache/beam/commit/6cdf35674a47ae92d9c836e0eb8fe1ffc583cbe4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cdf356) will **decrease** coverage by `0.80%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 5c2264f differs from pull request most recent head d4a4d4b. Consider uploading reports for the commit d4a4d4b to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #14244 +/- ##
==========================================
- Coverage 83.66% 82.86% -0.81%
==========================================
Files 446 466 +20
Lines 59239 57587 -1652
==========================================
- Hits 49564 47720 -1844
- Misses 9675 9867 +192
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...dks/python/apache\_beam/ml/gcp/naturallanguageml.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sLnB5) | | |
| [...sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | | |
| [...ython/apache\_beam/io/gcp/bigquery\_read\_internal.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3JlYWRfaW50ZXJuYWwucHk=) | | |
| [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | | |
| [...apache\_beam/examples/cookbook/custom\_ptransform.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY3VzdG9tX3B0cmFuc2Zvcm0ucHk=) | | |
| [.../build/srcs/sdks/python/apache\_beam/tools/utils.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvdXRpbHMucHk=) | | |
| [...build/srcs/sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | | |
| [...rcs/sdks/python/apache\_beam/runners/direct/util.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdXRpbC5weQ==) | | |
| [...he\_beam/portability/api/external\_transforms\_pb2.py](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2V4dGVybmFsX3RyYW5zZm9ybXNfcGIyLnB5) | | |
| ... and [902 more](https://codecov.io/gh/apache/beam/pull/14244/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cdf356...d4a4d4b](https://codecov.io/gh/apache/beam/pull/14244?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-816429265
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-812073870
@dennisylyung - Thank you for your contribution. Is this PR ready for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-813116210
> @dennisylyung - Thank you for your contribution. Is this PR ready for review?
Not yet sorry.
I am implementing for the aws2 module too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dennisylyung commented on pull request #14244: [BEAM-11907] SQS source with full functionality
Posted by GitBox <gi...@apache.org>.
dennisylyung commented on pull request #14244:
URL: https://github.com/apache/beam/pull/14244#issuecomment-825395334
Run Java_Examples_Dataflow_Java11 PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org