You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/25 19:45:23 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #12682: [BEAM-9547] Add a few more DataFrame operations
TheNeuralBit opened a new pull request #12682:
URL: https://github.com/apache/beam/pull/12682
- Combining Series with & and |
- Projecting multiple columns in __getitem__
- Dataframe.drop
R: @robertwb
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------



See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12682:
URL: https://github.com/apache/beam/pull/12682#discussion_r476707154
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -348,6 +358,8 @@ def aggregate(self, func, axis=0, *args, **kwargs):
'order-sensitive')
diff = frame_base.wont_implement_method('order-sensitive')
+ drop = frame_base._elementwise_method('drop')
Review comment:
Ok we can just drop that commit for now. I'm not actually using drop in the demo since `__getitem__` is cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-681025127
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82243 +724
Branches 9185 9297 +112
==========================================
+ Hits 28109 28229 +120
- Misses 52987 53591 +604
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.98% <0.00%> (-1.39%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [dataframe/transforms\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZGF0YWZyYW1lL3RyYW5zZm9ybXNfdGVzdC5weQ==) | `25.19% <0.00%> (-0.01%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...ad1e0b7](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12682:
URL: https://github.com/apache/beam/pull/12682
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82230 +711
Branches 9185 9295 +110
==========================================
+ Hits 28109 28227 +118
- Misses 52987 53580 +593
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [runners/common.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy9jb21tb24ucHk=) | `27.50% <0.00%> (ø)` | |
| [examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `15.70% <0.00%> (ø)` | |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...ad1e0b7](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12682:
URL: https://github.com/apache/beam/pull/12682#discussion_r476699247
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -348,6 +358,8 @@ def aggregate(self, func, axis=0, *args, **kwargs):
'order-sensitive')
diff = frame_base.wont_implement_method('order-sensitive')
+ drop = frame_base._elementwise_method('drop')
Review comment:
drop has a lot of arguments, not all of them work with elementwise.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -213,7 +213,14 @@ def update(self, other):
DeferredSeries,
'__i%s__' % base,
frame_base._elementwise_method('__i%s__' % base, inplace=True))
-for name in ['__lt__', '__le__', '__gt__', '__ge__', '__eq__', '__ne__']:
+for name in ['__lt__',
+ '__le__',
+ '__gt__',
+ '__ge__',
+ '__eq__',
+ '__ne__',
+ '__or__',
Review comment:
Add `__ior__` and `__iand__` as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82243 +724
Branches 9185 9297 +112
==========================================
+ Hits 28109 28229 +120
- Misses 52987 53591 +604
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.98% <0.00%> (-1.39%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [dataframe/transforms\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZGF0YWZyYW1lL3RyYW5zZm9ybXNfdGVzdC5weQ==) | `25.19% <0.00%> (-0.01%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...ad1e0b7](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12682:
URL: https://github.com/apache/beam/pull/12682#discussion_r477464890
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -348,6 +358,8 @@ def aggregate(self, func, axis=0, *args, **kwargs):
'order-sensitive')
diff = frame_base.wont_implement_method('order-sensitive')
+ drop = frame_base._elementwise_method('drop')
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12682:
URL: https://github.com/apache/beam/pull/12682#discussion_r477466005
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -213,7 +213,14 @@ def update(self, other):
DeferredSeries,
'__i%s__' % base,
frame_base._elementwise_method('__i%s__' % base, inplace=True))
-for name in ['__lt__', '__le__', '__gt__', '__ge__', '__eq__', '__ne__']:
+for name in ['__lt__',
+ '__le__',
+ '__gt__',
+ '__ge__',
+ '__eq__',
+ '__ne__',
+ '__or__',
Review comment:
Moved `or` and `and` so that `__ior__` and `__iand__` get defined
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82230 +711
Branches 9185 9295 +110
==========================================
+ Hits 28109 28227 +118
- Misses 52987 53580 +593
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [runners/common.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy9jb21tb24ucHk=) | `27.50% <0.00%> (ø)` | |
| [examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `15.70% <0.00%> (ø)` | |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...f890149](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82243 +724
Branches 9185 9297 +112
==========================================
+ Hits 28109 28229 +120
- Misses 52987 53591 +604
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.98% <0.00%> (-1.39%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [dataframe/transforms\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZGF0YWZyYW1lL3RyYW5zZm9ybXNfdGVzdC5weQ==) | `25.19% <0.00%> (-0.01%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...ad1e0b7](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12682: [BEAM-9547] Add a few more DataFrame operations
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12682:
URL: https://github.com/apache/beam/pull/12682#issuecomment-680243372
# [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=h1) Report
> Merging [#12682](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3e80a2113a4e06c9c1efe566b3db872cb96c8e91?el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12682 +/- ##
==========================================
- Coverage 34.48% 34.32% -0.16%
==========================================
Files 685 694 +9
Lines 81519 82230 +711
Branches 9185 9295 +110
==========================================
+ Hits 28109 28227 +118
- Misses 52987 53580 +593
Partials 423 423
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [typehints/typecheck\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVja190ZXN0X3B5My5weQ==) | `31.54% <0.00%> (-16.00%)` | :arrow_down: |
| [typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | `29.44% <0.00%> (-6.18%)` | :arrow_down: |
| [runners/worker/opcounters.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvb3Bjb3VudGVycy5weQ==) | `33.81% <0.00%> (-0.87%)` | :arrow_down: |
| [pipeline.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cGlwZWxpbmUucHk=) | `22.04% <0.00%> (-0.28%)` | :arrow_down: |
| [io/gcp/bigquery\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3F1ZXJ5X3Rlc3QucHk=) | `27.39% <0.00%> (-0.18%)` | :arrow_down: |
| [options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-b3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `52.99% <0.00%> (-0.16%)` | :arrow_down: |
| [transforms/ptransform\_test.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9wdHJhbnNmb3JtX3Rlc3QucHk=) | `18.37% <0.00%> (-0.09%)` | :arrow_down: |
| [transforms/core.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9jb3JlLnB5) | `36.79% <0.00%> (-0.05%)` | :arrow_down: |
| [runners/common.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-cnVubmVycy9jb21tb24ucHk=) | `27.50% <0.00%> (ø)` | |
| [examples/snippets/snippets.py](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHMucHk=) | `15.70% <0.00%> (ø)` | |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/12682/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=footer). Last update [e6db419...f890149](https://codecov.io/gh/apache/beam/pull/12682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org