You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/18 05:14:37 UTC
[GitHub] [beam] pabloem opened a new pull request #11153: Adding a new
module for FnApiRunner
pabloem opened a new pull request #11153: Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395897316
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -269,8 +269,8 @@ def visit_transform(self, transform_node):
pcoll.element_type, transform_node.full_label)
key_type, value_type = pcoll.element_type.tuple_types
if transform_node.outputs:
- from apache_beam.runners.portability.fn_api_runner_transforms import \
- only_element
+ from apache_beam.runners.portability.fn_api_runner.transforms \
Review comment:
I meant that utilities like only_element should probably be in a more common place rather than imported elsewhere from here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600741439
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600435252
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600783232
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395889275
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -269,8 +269,8 @@ def visit_transform(self, transform_node):
pcoll.element_type, transform_node.full_label)
key_type, value_type = pcoll.element_type.tuple_types
if transform_node.outputs:
- from apache_beam.runners.portability.fn_api_runner_transforms import \
- only_element
+ from apache_beam.runners.portability.fn_api_runner.transforms \
Review comment:
You mean the newly named `translations` module may need to exist in `apache_beam/utils`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600749448
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-601938785
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395885958
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -78,12 +78,12 @@
from apache_beam.runners import pipeline_context
from apache_beam.runners import runner
from apache_beam.runners.portability import artifact_service
-from apache_beam.runners.portability import fn_api_runner_transforms
from apache_beam.runners.portability import portable_metrics
-from apache_beam.runners.portability.fn_api_runner_transforms import create_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import only_element
-from apache_beam.runners.portability.fn_api_runner_transforms import split_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import unique_name
+from apache_beam.runners.portability.fn_api_runner import transforms
Review comment:
Seeing this unqualified makes me realize that it's ambiguous with the notion of a PTransform (and the apache_beam.transforms package). Maybe we should call it `optimizations` or `translations`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-601492165
@robertwb PTAL - I worry that there'll be merge conflicts soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600741147
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395888936
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -78,12 +78,12 @@
from apache_beam.runners import pipeline_context
from apache_beam.runners import runner
from apache_beam.runners.portability import artifact_service
-from apache_beam.runners.portability import fn_api_runner_transforms
from apache_beam.runners.portability import portable_metrics
-from apache_beam.runners.portability.fn_api_runner_transforms import create_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import only_element
-from apache_beam.runners.portability.fn_api_runner_transforms import split_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import unique_name
+from apache_beam.runners.portability.fn_api_runner import transforms
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395888936
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -78,12 +78,12 @@
from apache_beam.runners import pipeline_context
from apache_beam.runners import runner
from apache_beam.runners.portability import artifact_service
-from apache_beam.runners.portability import fn_api_runner_transforms
from apache_beam.runners.portability import portable_metrics
-from apache_beam.runners.portability.fn_api_runner_transforms import create_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import only_element
-from apache_beam.runners.portability.fn_api_runner_transforms import split_buffer_id
-from apache_beam.runners.portability.fn_api_runner_transforms import unique_name
+from apache_beam.runners.portability.fn_api_runner import transforms
Review comment:
Done. Went with translations. But I also like optimizations... Maybe I'll rename if it becomes a little bit more optimizations than translations down the road.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600740277
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-601923625
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem merged pull request #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11153: [BEAM-9537] Adding a new
module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#issuecomment-600865229
I've started rebasing https://github.com/apache/beam/pull/10291 to keep it up to date. Since the module refactoring is likely to make lots of trouble merging, I'm getting it out of the way early.
r: @robertwb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11153:
[BEAM-9537] Adding a new module for FnApiRunner
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11153: [BEAM-9537] Adding a new module for FnApiRunner
URL: https://github.com/apache/beam/pull/11153#discussion_r395884481
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -269,8 +269,8 @@ def visit_transform(self, transform_node):
pcoll.element_type, transform_node.full_label)
key_type, value_type = pcoll.element_type.tuple_types
if transform_node.outputs:
- from apache_beam.runners.portability.fn_api_runner_transforms import \
- only_element
+ from apache_beam.runners.portability.fn_api_runner.transforms \
Review comment:
You don't have to do this in this PR, but this should probably be in apache_beam/utils.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services