You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/17 19:20:30 UTC
[GitHub] [beam] KevinGG opened a new pull request #11147: [BEAM-7923]
Support dict and iterable PCollections in show
KevinGG opened a new pull request #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147
1. Added support of dict and iterable of PCollections as *args in the
show API.
2. Added typehints for show.
3. Added tests for the new support.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] KevinGG commented on a change in pull request #11147:
[BEAM-7923] Support dict and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#discussion_r393922585
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -276,6 +284,21 @@ def show(*pcolls, **configs):
# PCollection `square` and PCollection `cube`, then visualizes them.
show(square, cube)
"""
+ flatten_pcolls = []
+ for pcoll_container in pcolls:
+ if isinstance(pcoll_container, dict):
+ flatten_pcolls.extend(pcoll_container.values())
+ elif isinstance(pcoll_container, beam.pvalue.PCollection):
+ flatten_pcolls.append(pcoll_container)
+ else:
+ try:
+ flatten_pcolls.extend(iter(pcoll_container))
+ except TypeError:
+ _LOGGER.warning(
+ 'The given pcoll %s is not a dict, an iterable or a PCollection.',
+ pcoll_container)
+ return
Review comment:
Yes, we could raise a ValueError. Changing it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600339328
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600293642
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600317593
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600286040
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11147:
[BEAM-7923] Support dict and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#discussion_r393920308
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -276,6 +284,21 @@ def show(*pcolls, **configs):
# PCollection `square` and PCollection `cube`, then visualizes them.
show(square, cube)
"""
+ flatten_pcolls = []
+ for pcoll_container in pcolls:
+ if isinstance(pcoll_container, dict):
+ flatten_pcolls.extend(pcoll_container.values())
+ elif isinstance(pcoll_container, beam.pvalue.PCollection):
+ flatten_pcolls.append(pcoll_container)
+ else:
+ try:
+ flatten_pcolls.extend(iter(pcoll_container))
+ except TypeError:
+ _LOGGER.warning(
+ 'The given pcoll %s is not a dict, an iterable or a PCollection.',
+ pcoll_container)
+ return
Review comment:
It may be good to throw a ValueError or the TypeError? WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] KevinGG commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
KevinGG commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600283827
Changed to raise ValueError when input is invalid for `show`, also rebased to current HEAD to pick up most recent changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600317504
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600339261
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem merged pull request #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] KevinGG commented on issue #11147: [BEAM-7923] Support dict
and iterable PCollections in show
Posted by GitBox <gi...@apache.org>.
KevinGG commented on issue #11147: [BEAM-7923] Support dict and iterable PCollections in show
URL: https://github.com/apache/beam/pull/11147#issuecomment-600253084
Formatted with yapf.
Lint passed locally.
R: @pabloem
R: @rohdesamuel
R: @davidyan74
PTAL, thx!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services