You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "robertwb (via GitHub)" <gi...@apache.org> on 2023/03/22 00:40:07 UTC
[GitHub] [beam] robertwb opened a new pull request, #25919: Allow PTransforms to be applied directly to dataframes.
robertwb opened a new pull request, #25919:
URL: https://github.com/apache/beam/pull/25919
This'll be especially handy for applying RunInference.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #25919:
URL: https://github.com/apache/beam/pull/25919#issuecomment-1481713006
No problem. And, yes, the proto changes were merged separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25919:
URL: https://github.com/apache/beam/pull/25919#discussion_r1144959615
##########
sdks/python/gen_protos.py:
##########
@@ -123,7 +123,19 @@ def generate_urn_files(out_dir, api_path):
This is executed at build time rather than dynamically on import to ensure
that it is compatible with static type checkers like mypy.
"""
- from google._upb import _message
+ try:
+ from google._upb import _message
+ list_types = (
+ list,
+ _message.RepeatedScalarContainer,
+ _message.RepeatedCompositeContainer,
+ ) # pylint: disable=c-extension-no-member
+ except ImportError:
+ from google.protobuf.internal import containers
+ list_types = (
+ list,
+ containers.RepeatedScalarFieldContainer,
+ containers.RepeatedCompositeFieldContainer)
Review Comment:
i think this change is part of https://github.com/apache/beam/pull/25914. @robertwb is this change required here, if not can we continue discussion on https://github.com/apache/beam/pull/25914?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #25919:
URL: https://github.com/apache/beam/pull/25919#issuecomment-1478767911
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #25919:
URL: https://github.com/apache/beam/pull/25919#issuecomment-1481625347
Just checking whether "I think this is fine" is an LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25919:
URL: https://github.com/apache/beam/pull/25919#issuecomment-1480345450
## [Codecov](https://codecov.io/gh/apache/beam/pull/25919?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#25919](https://codecov.io/gh/apache/beam/pull/25919?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (029451b) into [master](https://codecov.io/gh/apache/beam/commit/fcf51fb7d98968d55e96981508d476ec7cd49400?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fcf51fb) will **increase** coverage by `0.00%`.
> The diff coverage is `90.90%`.
```diff
@@ Coverage Diff @@
## master #25919 +/- ##
=======================================
Coverage 71.41% 71.41%
=======================================
Files 778 778
Lines 102420 102441 +21
=======================================
+ Hits 73146 73162 +16
- Misses 27818 27823 +5
Partials 1456 1456
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `79.96% <90.90%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/25919?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/25919?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `95.31% <90.90%> (-0.03%)` | :arrow_down: |
... and [8 files with indirect coverage changes](https://codecov.io/gh/apache/beam/pull/25919/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #25919:
URL: https://github.com/apache/beam/pull/25919#discussion_r1144933210
##########
sdks/python/gen_protos.py:
##########
@@ -123,7 +123,19 @@ def generate_urn_files(out_dir, api_path):
This is executed at build time rather than dynamically on import to ensure
that it is compatible with static type checkers like mypy.
"""
- from google._upb import _message
+ try:
+ from google._upb import _message
+ list_types = (
+ list,
+ _message.RepeatedScalarContainer,
+ _message.RepeatedCompositeContainer,
+ ) # pylint: disable=c-extension-no-member
+ except ImportError:
+ from google.protobuf.internal import containers
+ list_types = (
+ list,
+ containers.RepeatedScalarFieldContainer,
+ containers.RepeatedCompositeFieldContainer)
Review Comment:
Trying to follow why we need this - is this from having an old protobuf installed? It looks like we previously used `google.protobuf.pyext._message` before https://github.com/apache/beam/commit/5cb1711c39c2ebc3005363f938ca4a4eee774333, is there a reason we're now using `google.protobuf.internal.containers`?
cc/ @AnandInguva @tvalentyn
##########
sdks/python/apache_beam/dataframe/frames.py:
##########
@@ -5394,6 +5396,24 @@ def func(df, *args, **kwargs):
frame_base._elementwise_method(inplace_name, inplace=True,
base=pd.DataFrame))
+# Allow dataframe | SchemaTransform
+def _create_maybe_elementwise_or(base):
+ elementwise = frame_base._elementwise_method(
+ '__or__', restrictions={'level': None}, base=base)
+
+ def _maybe_elementwise_or(self, right):
+ if isinstance(right, PTransform):
+ return convert.to_dataframe(convert.to_pcollection(self) | right)
Review Comment:
I think this is fine, it is worth calling out that we're opening users up to doing an inefficient thing where they go:
```
df = convert.to_dataframe(pc)
df2 = df | MyDfOperation()
df3 = df2 | MySchemaTransform() | MySchemaTransform2() | MySchemaTransform3()
result = convert.to_dataframe(df3)
```
which would be (much?) more efficient written as:
```
df = convert.to_dataframe(pc)
df2 = df | MyDfOperation()
result = convert.to_dataframe(df2) | MySchemaTransform() | MySchemaTransform2() | MySchemaTransform3()
```
because this avoids the repeated `to_dataframe`/`to_pcollection` transition. The former is probably more natural though, especially if you have real df operations mixed in there even if its less efficient. I think the user experience still trumps the efficiency loss, but it might be something we want to doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25919:
URL: https://github.com/apache/beam/pull/25919#issuecomment-1478768416
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #25919:
URL: https://github.com/apache/beam/pull/25919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25919:
URL: https://github.com/apache/beam/pull/25919#discussion_r1145462923
##########
sdks/python/gen_protos.py:
##########
@@ -123,7 +123,19 @@ def generate_urn_files(out_dir, api_path):
This is executed at build time rather than dynamically on import to ensure
that it is compatible with static type checkers like mypy.
"""
- from google._upb import _message
+ try:
+ from google._upb import _message
+ list_types = (
+ list,
+ _message.RepeatedScalarContainer,
+ _message.RepeatedCompositeContainer,
+ ) # pylint: disable=c-extension-no-member
+ except ImportError:
+ from google.protobuf.internal import containers
+ list_types = (
+ list,
+ containers.RepeatedScalarFieldContainer,
+ containers.RepeatedCompositeFieldContainer)
Review Comment:
Sorry, this was part of another PR. That's been merged; rebased this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25919:
URL: https://github.com/apache/beam/pull/25919#discussion_r1145462426
##########
sdks/python/apache_beam/dataframe/frames.py:
##########
@@ -5394,6 +5396,24 @@ def func(df, *args, **kwargs):
frame_base._elementwise_method(inplace_name, inplace=True,
base=pd.DataFrame))
+# Allow dataframe | SchemaTransform
+def _create_maybe_elementwise_or(base):
+ elementwise = frame_base._elementwise_method(
+ '__or__', restrictions={'level': None}, base=base)
+
+ def _maybe_elementwise_or(self, right):
+ if isinstance(right, PTransform):
+ return convert.to_dataframe(convert.to_pcollection(self) | right)
Review Comment:
Added a note. Once we push the batching stuff through, there could be little to no overhead here, but that's future work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #25919: Allow PTransforms to be applied directly to dataframes.
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25919:
URL: https://github.com/apache/beam/pull/25919#discussion_r1144974776
##########
sdks/python/gen_protos.py:
##########
@@ -123,7 +123,19 @@ def generate_urn_files(out_dir, api_path):
This is executed at build time rather than dynamically on import to ensure
that it is compatible with static type checkers like mypy.
"""
- from google._upb import _message
+ try:
+ from google._upb import _message
+ list_types = (
+ list,
+ _message.RepeatedScalarContainer,
+ _message.RepeatedCompositeContainer,
+ ) # pylint: disable=c-extension-no-member
+ except ImportError:
+ from google.protobuf.internal import containers
+ list_types = (
+ list,
+ containers.RepeatedScalarFieldContainer,
+ containers.RepeatedCompositeFieldContainer)
Review Comment:
I think even with older protobuf, we would need `google.protobuf.pyext._message` when `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION==cpp` but we might need `google.protobuf.internal.containers` when `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python`.
commented at https://github.com/apache/beam/pull/25914#discussion_r1144970641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org