You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/03 23:22:25 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #14140: [BEAM-9547] Add groupby doctests
TheNeuralBit opened a new pull request #14140:
URL: https://github.com/apache/beam/pull/14140
Adds a new test method in pandas_doctests_test.py that runs the doctests from groupby objects.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (e80d490) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.01%
==========================================
Files 469 469
Lines 58473 58485 +12
==========================================
+ Hits 48565 48570 +5
- Misses 9908 9915 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/examples/snippets/transforms/elementwise/map.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9tYXAucHk=) | | |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | | |
| [...y38/build/srcs/sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | | |
| [...examples/snippets/transforms/elementwise/values.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS92YWx1ZXMucHk=) | | |
| [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | | |
| [...ython/apache\_beam/io/aws/clients/s3/fake\_client.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvZmFrZV9jbGllbnQucHk=) | | |
| [...rcs/sdks/python/apache\_beam/io/aws/s3filesystem.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL3MzZmlsZXN5c3RlbS5weQ==) | | |
| [...ks/python/apache\_beam/runners/worker/sideinputs.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2lkZWlucHV0cy5weQ==) | | |
| [.../python/apache\_beam/testing/benchmarks/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL19faW5pdF9fLnB5) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...e80d490](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (6bc418e) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.02%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48575 +10
- Misses 9908 9918 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/dataflow/internal/names.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9uYW1lcy5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/io/gcp/spanner.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3NwYW5uZXIucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/runners/sdf\_utils.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9zZGZfdXRpbHMucHk=) | | |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | | |
| [.../sdks/python/apache\_beam/io/external/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvZ2NwL3B1YnN1Yi5weQ==) | | |
| [...am/examples/snippets/transforms/aggregation/top.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi90b3AucHk=) | | |
| [...ransforms/transforms\_keyword\_only\_args\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmFuc2Zvcm1zX2tleXdvcmRfb25seV9hcmdzX3Rlc3RfcHkzLnB5) | | |
| [...pache\_beam/runners/portability/artifact\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hcnRpZmFjdF9zZXJ2aWNlLnB5) | | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | | |
| [...e\_beam/runners/portability/abstract\_job\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hYnN0cmFjdF9qb2Jfc2VydmljZS5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...039551f](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587723645
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
Review comment:
1. Done.
2. You're right on both counts. That's actually why I recently switched our pandas dependency to restrict the next minor version (and made myself the owner of that dep so I'll know when we need to upgrade). In general bugfix releases shouldn't break this suite, but it's likely minor releases will, as they can add new functionality that we don't support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (039551f) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.05% -0.01%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48580 +15
- Misses 9908 9913 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/dataflow/internal/names.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9uYW1lcy5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/io/gcp/spanner.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3NwYW5uZXIucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/runners/sdf\_utils.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9zZGZfdXRpbHMucHk=) | | |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | | |
| [.../sdks/python/apache\_beam/io/external/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvZ2NwL3B1YnN1Yi5weQ==) | | |
| [...am/examples/snippets/transforms/aggregation/top.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi90b3AucHk=) | | |
| [...ransforms/transforms\_keyword\_only\_args\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmFuc2Zvcm1zX2tleXdvcmRfb25seV9hcmdzX3Rlc3RfcHkzLnB5) | | |
| [...pache\_beam/runners/portability/artifact\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hcnRpZmFjdF9zZXJ2aWNlLnB5) | | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | | |
| [...e\_beam/runners/portability/abstract\_job\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hYnN0cmFjdF9qb2Jfc2VydmljZS5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...039551f](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #14140:
URL: https://github.com/apache/beam/pull/14140
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (e80d490) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.01%
==========================================
Files 469 469
Lines 58473 58485 +12
==========================================
+ Hits 48565 48570 +5
- Misses 9908 9915 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...srcs/sdks/python/apache\_beam/transforms/trigger.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmlnZ2VyLnB5) | | |
| [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | | |
| [.../apache\_beam/runners/direct/transform\_evaluator.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdHJhbnNmb3JtX2V2YWx1YXRvci5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/tools/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvX19pbml0X18ucHk=) | | |
| [...8/build/srcs/sdks/python/apache\_beam/io/kinesis.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8va2luZXNpcy5weQ==) | | |
| [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | | |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/coders/avro\_record.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2F2cm9fcmVjb3JkLnB5) | | |
| [...am/portability/api/standard\_window\_fns\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3N0YW5kYXJkX3dpbmRvd19mbnNfcGIyX3VybnMucHk=) | | |
| [...ython/apache\_beam/io/gcp/experimental/spannerio.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2V4cGVyaW1lbnRhbC9zcGFubmVyaW8ucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...e80d490](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587900258
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
Review comment:
Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r586915590
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
+ pd.core.groupby.groupby,
+ use_beam=False,
+ wont_implement_ok={
+ 'pandas.core.groupby.groupby.GroupBy.head': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.tail': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.cumcount': ['*'],
+ },
+ not_implemented_ok={
+ 'pandas.core.groupby.groupby.GroupBy.describe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.sample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.quantile': ['*'],
+ 'pandas.core.groupby.groupby.BaseGroupBy.pipe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': [
+ "df.groupby('A', as_index=False).nth(1)",
+ ],
+ },
+ skip={
+ # Uses iloc to mutate a DataFrame
+ 'pandas.core.groupby.groupby.GroupBy.resample': [
+ 'df.iloc[2, 0] = 5',
+ 'df',
+ ],
+ # TODO: Raise wont implement for list passed as a grouping column
+ # Currently raises unhashable type: list
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': [
+ 'df.groupby(["A", [1,1,2,3,2,1]]).ngroup()'
+ ],
+ })
+ self.assertEqual(result.failed, 0)
+
+ result = doctests.testmod(
+ pd.core.groupby.generic,
+ use_beam=False,
+ wont_implement_ok={
+ # Returns an array by default, not a Series. WontImplement
+ # (non-deferred)
+ 'pandas.core.groupby.generic.SeriesGroupBy.unique': ['*'],
+ # TODO: Is take actually deprecated?
+ 'pandas.core.groupby.generic.DataFrameGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.nsmallest': [
+ "s.nsmallest(3, keep='last')",
Review comment:
For my education:
1. Does ['*'] mean something like exclude all tests in this class?
2. I am seeing `"s.nsmallest(3, keep='last')"` in pandas/core/series.py. How does it relate to `pd.core.groupby.generic` ?
https://github.com/pandas-dev/pandas/blob/440a7de1c328aede3f566830c989922d46c254a8/pandas/core/series.py#L3686
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (6bc418e) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.02%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48575 +10
- Misses 9908 9918 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | | |
| [...apache\_beam/testing/benchmarks/nexmark/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvX19pbml0X18ucHk=) | | |
| [...ache\_beam/testing/benchmarks/chicago\_taxi/setup.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9zZXR1cC5weQ==) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query0.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTAucHk=) | | |
| [...ples/snippets/transforms/aggregation/groupbykey.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9ncm91cGJ5a2V5LnB5) | | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | | |
| [...dks/python/apache\_beam/transforms/create\_source.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jcmVhdGVfc291cmNlLnB5) | | |
| [...examples/snippets/transforms/aggregation/sample.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9zYW1wbGUucHk=) | | |
| [...nchmarks/chicago\_taxi/tfdv\_analyze\_and\_validate.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS90ZmR2X2FuYWx5emVfYW5kX3ZhbGlkYXRlLnB5) | | |
| [...8/build/srcs/sdks/python/apache\_beam/utils/urns.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdXJucy5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...6bc418e](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r586910596
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
Review comment:
1. Could you please add a docsting for `doctests.testmod()`?
For example, it's not clear what `use_beam` means and what parameters are allowed.
2. Do I understand correctly that this test will run the repl commands in pd.core.groupby.groupby (Pandas code)? If so, do we expect to be broken by Pandas adding new functionality that Beam Dataframe doesn't support yet?
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
+ pd.core.groupby.groupby,
+ use_beam=False,
+ wont_implement_ok={
+ 'pandas.core.groupby.groupby.GroupBy.head': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.tail': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.cumcount': ['*'],
+ },
+ not_implemented_ok={
+ 'pandas.core.groupby.groupby.GroupBy.describe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.sample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.quantile': ['*'],
+ 'pandas.core.groupby.groupby.BaseGroupBy.pipe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': [
+ "df.groupby('A', as_index=False).nth(1)",
+ ],
+ },
+ skip={
+ # Uses iloc to mutate a DataFrame
+ 'pandas.core.groupby.groupby.GroupBy.resample': [
+ 'df.iloc[2, 0] = 5',
+ 'df',
+ ],
+ # TODO: Raise wont implement for list passed as a grouping column
+ # Currently raises unhashable type: list
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': [
+ 'df.groupby(["A", [1,1,2,3,2,1]]).ngroup()'
+ ],
+ })
+ self.assertEqual(result.failed, 0)
+
+ result = doctests.testmod(
+ pd.core.groupby.generic,
+ use_beam=False,
+ wont_implement_ok={
+ # Returns an array by default, not a Series. WontImplement
+ # (non-deferred)
+ 'pandas.core.groupby.generic.SeriesGroupBy.unique': ['*'],
+ # TODO: Is take actually deprecated?
+ 'pandas.core.groupby.generic.DataFrameGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.nsmallest': [
+ "s.nsmallest(3, keep='last')",
Review comment:
1. Does ['*'] mean something like exclude all tests in this class?
2. I am seeing `"s.nsmallest(3, keep='last')"` in pandas/core/series.py. How does it relate to `pd.core.groupby.generic` ?
https://github.com/pandas-dev/pandas/blob/440a7de1c328aede3f566830c989922d46c254a8/pandas/core/series.py#L3686
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (039551f) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.05% -0.01%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48580 +15
- Misses 9908 9913 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/dataflow/internal/clients/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9jbGllbnRzL19faW5pdF9fLnB5) | | |
| [.../examples/snippets/transforms/aggregation/count.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb3VudC5weQ==) | | |
| [...thon/apache\_beam/runners/worker/channel\_factory.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvY2hhbm5lbF9mYWN0b3J5LnB5) | | |
| [...dks/python/apache\_beam/io/external/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvZ2NwL19faW5pdF9fLnB5) | | |
| [...e\_beam/runners/portability/abstract\_job\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hYnN0cmFjdF9qb2Jfc2VydmljZS5weQ==) | | |
| [...on/apache\_beam/io/gcp/bigquery\_io\_read\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX3JlYWRfcGlwZWxpbmUucHk=) | | |
| [...nternal/clients/dataflow/dataflow\_v1b3\_messages.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9jbGllbnRzL2RhdGFmbG93L2RhdGFmbG93X3YxYjNfbWVzc2FnZXMucHk=) | | |
| [...uild/srcs/sdks/python/apache\_beam/internal/util.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvdXRpbC5weQ==) | | |
| [...ild/srcs/sdks/python/apache\_beam/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tZXRyaWMucHk=) | | |
| [...d/srcs/sdks/python/apache\_beam/testing/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9fX2luaXRfXy5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...039551f](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (039551f) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.05% -0.01%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48580 +15
- Misses 9908 9913 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/testing/benchmarks/chicago\_taxi/process\_tfma.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2NoaWNhZ29fdGF4aS9wcm9jZXNzX3RmbWEucHk=) | | |
| [...ython/apache\_beam/typehints/decorators\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2RlY29yYXRvcnNfdGVzdF9weTMucHk=) | | |
| [...tes/tox/py38/build/srcs/sdks/python/test\_config.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vdGVzdF9jb25maWcucHk=) | | |
| [...uild/srcs/sdks/python/apache\_beam/metrics/cells.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9jZWxscy5weQ==) | | |
| [...s/sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | | |
| [...s/python/apache\_beam/runners/worker/log\_handler.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvbG9nX2hhbmRsZXIucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query5.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTUucHk=) | | |
| [...s/python/apache\_beam/typehints/sharded\_key\_type.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NoYXJkZWRfa2V5X3R5cGUucHk=) | | |
| [.../apache\_beam/testing/benchmarks/nexmark/monitor.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbW9uaXRvci5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...039551f](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790841263
Thanks for the quick review. The CI failures _should_ be addressed now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587901043
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
+ pd.core.groupby.groupby,
+ use_beam=False,
+ wont_implement_ok={
+ 'pandas.core.groupby.groupby.GroupBy.head': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.tail': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.cumcount': ['*'],
+ },
+ not_implemented_ok={
+ 'pandas.core.groupby.groupby.GroupBy.describe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.sample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.quantile': ['*'],
+ 'pandas.core.groupby.groupby.BaseGroupBy.pipe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': [
+ "df.groupby('A', as_index=False).nth(1)",
+ ],
+ },
+ skip={
+ # Uses iloc to mutate a DataFrame
+ 'pandas.core.groupby.groupby.GroupBy.resample': [
+ 'df.iloc[2, 0] = 5',
+ 'df',
+ ],
+ # TODO: Raise wont implement for list passed as a grouping column
+ # Currently raises unhashable type: list
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': [
+ 'df.groupby(["A", [1,1,2,3,2,1]]).ngroup()'
+ ],
+ })
+ self.assertEqual(result.failed, 0)
+
+ result = doctests.testmod(
+ pd.core.groupby.generic,
+ use_beam=False,
+ wont_implement_ok={
+ # Returns an array by default, not a Series. WontImplement
+ # (non-deferred)
+ 'pandas.core.groupby.generic.SeriesGroupBy.unique': ['*'],
+ # TODO: Is take actually deprecated?
+ 'pandas.core.groupby.generic.DataFrameGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.nsmallest': [
+ "s.nsmallest(3, keep='last')",
Review comment:
I see. This explains it:
```In [10]: pandas.core.groupby.generic.SeriesGroupBy.nsmallest.__doc__ == pandas.core.series.Series.nsmallest.__doc__
Out[10]: True
```
I hope you don't have to deal with too many exclusions of the same method due to inheritance. A smarter match might help avoid repeats but perhaps not warranted now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587665489
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
+ pd.core.groupby.groupby,
+ use_beam=False,
+ wont_implement_ok={
+ 'pandas.core.groupby.groupby.GroupBy.head': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.tail': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.cumcount': ['*'],
+ },
+ not_implemented_ok={
+ 'pandas.core.groupby.groupby.GroupBy.describe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.sample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.quantile': ['*'],
+ 'pandas.core.groupby.groupby.BaseGroupBy.pipe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': [
+ "df.groupby('A', as_index=False).nth(1)",
+ ],
+ },
+ skip={
+ # Uses iloc to mutate a DataFrame
+ 'pandas.core.groupby.groupby.GroupBy.resample': [
+ 'df.iloc[2, 0] = 5',
+ 'df',
+ ],
+ # TODO: Raise wont implement for list passed as a grouping column
+ # Currently raises unhashable type: list
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': [
+ 'df.groupby(["A", [1,1,2,3,2,1]]).ngroup()'
+ ],
+ })
+ self.assertEqual(result.failed, 0)
+
+ result = doctests.testmod(
+ pd.core.groupby.generic,
+ use_beam=False,
+ wont_implement_ok={
+ # Returns an array by default, not a Series. WontImplement
+ # (non-deferred)
+ 'pandas.core.groupby.generic.SeriesGroupBy.unique': ['*'],
+ # TODO: Is take actually deprecated?
+ 'pandas.core.groupby.generic.DataFrameGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.nsmallest': [
+ "s.nsmallest(3, keep='last')",
Review comment:
1. Correct, except these are generally references to methods and their docstrings. It's implemented here: https://github.com/apache/beam/blob/05f520074b3a456f2ecf5df44f1602ca14bcb415/sdks/python/apache_beam/dataframe/doctests.py#L316-L320
2. Yeah I was confused by this too. It looks like some of these docstrings aren't actually testing the thing they say they do. Presumably due to some inheritance structure? It may behoove us to help clean up some of these examples upstream. (e.g. I just sent https://github.com/pandas-dev/pandas/pull/40207 for groupby.apply)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587901043
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -411,6 +411,96 @@ def test_indexing_tests(self):
})
self.assertEqual(result.failed, 0)
+ def test_groupby_tests(self):
+ result = doctests.testmod(
+ pd.core.groupby.groupby,
+ use_beam=False,
+ wont_implement_ok={
+ 'pandas.core.groupby.groupby.GroupBy.head': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.tail': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.cumcount': ['*'],
+ },
+ not_implemented_ok={
+ 'pandas.core.groupby.groupby.GroupBy.describe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.sample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.resample': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.quantile': ['*'],
+ 'pandas.core.groupby.groupby.BaseGroupBy.pipe': ['*'],
+ 'pandas.core.groupby.groupby.GroupBy.nth': [
+ "df.groupby('A', as_index=False).nth(1)",
+ ],
+ },
+ skip={
+ # Uses iloc to mutate a DataFrame
+ 'pandas.core.groupby.groupby.GroupBy.resample': [
+ 'df.iloc[2, 0] = 5',
+ 'df',
+ ],
+ # TODO: Raise wont implement for list passed as a grouping column
+ # Currently raises unhashable type: list
+ 'pandas.core.groupby.groupby.GroupBy.ngroup': [
+ 'df.groupby(["A", [1,1,2,3,2,1]]).ngroup()'
+ ],
+ })
+ self.assertEqual(result.failed, 0)
+
+ result = doctests.testmod(
+ pd.core.groupby.generic,
+ use_beam=False,
+ wont_implement_ok={
+ # Returns an array by default, not a Series. WontImplement
+ # (non-deferred)
+ 'pandas.core.groupby.generic.SeriesGroupBy.unique': ['*'],
+ # TODO: Is take actually deprecated?
+ 'pandas.core.groupby.generic.DataFrameGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.take': ['*'],
+ 'pandas.core.groupby.generic.SeriesGroupBy.nsmallest': [
+ "s.nsmallest(3, keep='last')",
Review comment:
I see. This explains it:
```
In [10]: pandas.core.groupby.generic.SeriesGroupBy.nsmallest.__doc__ == pandas.core.series.Series.nsmallest.__doc__
Out[10]: True
```
I hope you don't have to deal with too many exclusions of the same method due to inheritance. A smarter match might help avoid repeats but perhaps not warranted now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (e80d490) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.01%
==========================================
Files 469 469
Lines 58473 58485 +12
==========================================
+ Hits 48565 48570 +5
- Misses 9908 9915 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...\_beam/testing/benchmarks/nexmark/queries/query9.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTkucHk=) | | |
| [...examples/snippets/transforms/elementwise/kvswap.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9rdnN3YXAucHk=) | | |
| [...nippets/transforms/aggregation/groupintobatches.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9ncm91cGludG9iYXRjaGVzLnB5) | | |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | | |
| [...e\_beam/examples/complete/game/hourly\_team\_score.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9ob3VybHlfdGVhbV9zY29yZS5weQ==) | | |
| [...am/examples/snippets/transforms/elementwise/map.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9tYXAucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/examples/sql\_taxi.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc3FsX3RheGkucHk=) | | |
| [...sdks/python/apache\_beam/io/restriction\_trackers.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcmVzdHJpY3Rpb25fdHJhY2tlcnMucHk=) | | |
| [...sdks/python/apache\_beam/transforms/environments.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=) | | |
| [...ild/srcs/sdks/python/apache\_beam/utils/counters.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvY291bnRlcnMucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...e80d490](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587899821
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -666,3 +658,53 @@ def _run_patched(func, *args, **kwargs):
*args, extraglobs=extraglobs, optionflags=optionflags, **kwargs)
finally:
doctest.DocTestRunner = original_doc_test_runner
+
+
+def with_run_patched_docstring(target=None):
+ assert target is not None
+ def wrapper(fn):
+ fn.__doc__ = f"""Run all pandas doctests in the specified {target}.
+
+ Arguments `skip`, `wont_implement_ok`, `not_implemented_ok` are all in the
+ format::
+
+ {{
+ "module.Class.method": ['*'],
+ "module.Class.other_method": [
+ 'instance.other_method(bad_input)',
+ 'observe_result_of_bad_input()',
+ ],
+ }}
+
+ `'*'` indicates all examples should be matched, otherwise the list is a list
+ of specific input strings that should be matched.
+
+ All arguments are kwargs.
+
+ Args:
+ optionflags (int): Passed through to doctests.
+ extraglobs (Dict[str,Any]): Passed through to doctests.
+ use_beam (bool): If true, run a Beam pipeline with partitioned input to
+ verify the examples, else use PartitioningSession to simulate
+ distributed execution.
+ skip (Dict[str,str]): A set of examples to skip entirely.
+ wont_implement_ok (Dict[str,str]): A set of examples that are allowed to
+ raise WontImplementError.
+ not_implemented_ok (Dict[str,str]): A set of examples that are allowed to
+ raise NotImplementedError.
+
+ Returns:
+ ~doctest.TestResult: A doctest result describing the passed/failed tests.
+ """
+ return fn
+ return wrapper
+
+
+@with_run_patched_docstring(target="file")
Review comment:
What is the difference between file and module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (e80d490) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.01%
==========================================
Files 469 469
Lines 58473 58485 +12
==========================================
+ Hits 48565 48570 +5
- Misses 9908 9915 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ers/interactive/testing/integration/screen\_diff.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL2ludGVncmF0aW9uL3NjcmVlbl9kaWZmLnB5) | | |
| [...python/apache\_beam/examples/wordcount\_debugging.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RlYnVnZ2luZy5weQ==) | | |
| [...dks/python/apache\_beam/transforms/external\_java.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbF9qYXZhLnB5) | | |
| [.../apache\_beam/runners/dataflow/internal/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9fX2luaXRfXy5weQ==) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query6.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTYucHk=) | | |
| [...\_beam/testing/benchmarks/nexmark/queries/query5.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTUucHk=) | | |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | | |
| [...d/srcs/sdks/python/apache\_beam/internal/pickler.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvcGlja2xlci5weQ==) | | |
| [...python/apache\_beam/io/gcp/experimental/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2V4cGVyaW1lbnRhbC9fX2luaXRfXy5weQ==) | | |
| [...apache\_beam/examples/cookbook/custom\_ptransform.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY3VzdG9tX3B0cmFuc2Zvcm0ucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...e80d490](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (039551f) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.05% -0.01%
==========================================
Files 469 469
Lines 58473 58493 +20
==========================================
+ Hits 48565 48580 +15
- Misses 9908 9913 +5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...uild/srcs/sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | | |
| [...am/examples/snippets/transforms/aggregation/max.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9tYXgucHk=) | | |
| [...apache\_beam/examples/cookbook/custom\_ptransform.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY3VzdG9tX3B0cmFuc2Zvcm0ucHk=) | | |
| [...rcs/sdks/python/apache\_beam/io/aws/s3filesystem.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL3MzZmlsZXN5c3RlbS5weQ==) | | |
| [.../srcs/sdks/python/apache\_beam/examples/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvX19pbml0X18ucHk=) | | |
| [...apache\_beam/runners/dataflow/native\_io/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9uYXRpdmVfaW8vX19pbml0X18ucHk=) | | |
| [...nippets/transforms/aggregation/groupintobatches.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9ncm91cGludG9iYXRjaGVzLnB5) | | |
| [...py38/build/srcs/sdks/python/apache\_beam/version.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdmVyc2lvbi5weQ==) | | |
| [...hon/apache\_beam/runners/worker/worker\_pool\_main.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3Bvb2xfbWFpbi5weQ==) | | |
| [...srcs/sdks/python/apache\_beam/dataframe/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL19faW5pdF9fLnB5) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...039551f](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790152923
# [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=h1) Report
> Merging [#14140](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=desc) (e80d490) into [master](https://codecov.io/gh/apache/beam/commit/d378d57a568b79bde7c258e407cb0f3f5c1703a2?el=desc) (d378d57) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14140 +/- ##
==========================================
- Coverage 83.05% 83.04% -0.01%
==========================================
Files 469 469
Lines 58473 58485 +12
==========================================
+ Hits 48565 48570 +5
- Misses 9908 9915 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/examples/complete/game/game\_stats.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9nYW1lX3N0YXRzLnB5) | | |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | | |
| [.../build/srcs/sdks/python/apache\_beam/tools/utils.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvdXRpbHMucHk=) | | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | | |
| [...ache\_beam/runners/portability/local\_job\_service.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZS5weQ==) | | |
| [...amples/snippets/transforms/elementwise/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9fX2luaXRfXy5weQ==) | | |
| [...dks/python/apache\_beam/transforms/create\_source.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jcmVhdGVfc291cmNlLnB5) | | |
| [...hon/apache\_beam/portability/api/beam\_fn\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fZm5fYXBpX3BiMi5weQ==) | | |
| [...ld/srcs/sdks/python/apache\_beam/io/gcp/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL19faW5pdF9fLnB5) | | |
| [.../srcs/sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14140/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=footer). Last update [644cf47...6bc418e](https://codecov.io/gh/apache/beam/pull/14140?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #14140:
URL: https://github.com/apache/beam/pull/14140#issuecomment-790185036
Thanks, @TheNeuralBit ! Left a few questions, noting that several test suites/lint failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14140: [BEAM-9547] Add groupby doctests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14140:
URL: https://github.com/apache/beam/pull/14140#discussion_r587949924
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -666,3 +658,53 @@ def _run_patched(func, *args, **kwargs):
*args, extraglobs=extraglobs, optionflags=optionflags, **kwargs)
finally:
doctest.DocTestRunner = original_doc_test_runner
+
+
+def with_run_patched_docstring(target=None):
+ assert target is not None
+ def wrapper(fn):
+ fn.__doc__ = f"""Run all pandas doctests in the specified {target}.
+
+ Arguments `skip`, `wont_implement_ok`, `not_implemented_ok` are all in the
+ format::
+
+ {{
+ "module.Class.method": ['*'],
+ "module.Class.other_method": [
+ 'instance.other_method(bad_input)',
+ 'observe_result_of_bad_input()',
+ ],
+ }}
+
+ `'*'` indicates all examples should be matched, otherwise the list is a list
+ of specific input strings that should be matched.
+
+ All arguments are kwargs.
+
+ Args:
+ optionflags (int): Passed through to doctests.
+ extraglobs (Dict[str,Any]): Passed through to doctests.
+ use_beam (bool): If true, run a Beam pipeline with partitioned input to
+ verify the examples, else use PartitioningSession to simulate
+ distributed execution.
+ skip (Dict[str,str]): A set of examples to skip entirely.
+ wont_implement_ok (Dict[str,str]): A set of examples that are allowed to
+ raise WontImplementError.
+ not_implemented_ok (Dict[str,str]): A set of examples that are allowed to
+ raise NotImplementedError.
+
+ Returns:
+ ~doctest.TestResult: A doctest result describing the passed/failed tests.
+ """
+ return fn
+ return wrapper
+
+
+@with_run_patched_docstring(target="file")
Review comment:
They correspond to the [doctest](https://docs.python.org/3/library/doctest.html) methods of the same name. It looks like we don't actually use testfile at the moment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org