You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by greghogan <gi...@git.apache.org> on 2016/07/25 18:10:30 UTC
[GitHub] flink pull request #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/2294
[FLINK-4265] [dataset api] Add a NoOpOperator
Adds a NoOpOperator which is unwound in OperatorTranslation.translate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 4265_add_a_noopoperator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2294
----
commit cc7b9f199777f396d5d15b789480486278267117
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-07-25T18:05:56Z
[FLINK-4265] [dataset api] Add a NoOpOperator
Adds a NoOpOperator which is unwound in OperatorTranslation.translate.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2294
@fhueske @ggevay what are your thoughts on adding this as an `@Internal` operator?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2294
@greghogan I am not sure I completely understand this. If I see it correctly, the no-op operator is in the API, but gets removed during the translation?
It looks like a bit of a hacky approach to me. Can you try and explain how exactly it helps?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2294
@fhueske what is your analysis?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2294
@StephanEwen thoughts on how to proceed? Not looking to rush this.
With this simple PR I can remove the use of javassist's `ProxyFactory` from Gelly. Or we can defer consideration of this PR until 1.2.0 and I can apply the temporary 90% fix to Gelly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2294
Sorry for the late reply @greghogan.
+1 from me as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2294
The alternatives are much more hacky.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2294
This would replace `Delegate` and is much simpler. I placed it in the Java API package but it would not be visible to users through the `DataSet` API.
The ultimate goal is to implicitly reuse operations performing the same computation on the same input `DataSet`. The optimizer cannot do this without understanding the UDF configuration and logic. Two instances of a UDF may be
* incompatible, in which case a new result must be computed
* the same, in which case the old result can be reused
* different but compatible, in which case the UDF can merge the configurations and the new, shared result replaces the old result
Replacing the old result requires a wrapper. Using `ProxyFactory` in `Delegate` has limitations as documented in FLINK-4257. With a `NoOpOperator` we can perform the same function by appending and then ignoring a dummy POJO operator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2294: [FLINK-4265] [dataset api] Add a NoOpOperator
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2294
This is okay to merge from my side. Definitely seems easier than the `Delegate` trick.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---