You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by greghogan <gi...@git.apache.org> on 2016/05/25 11:55:38 UTC
[GitHub] flink pull request: [FLINK-3965] [gelly] Delegating GraphAlgorithm
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/2032
[FLINK-3965] [gelly] Delegating GraphAlgorithm
A DelegatingGraphAlgorithm wraps a GraphAlgorithm result with a delegating proxy object. The delegated object can be replaced when the same algorithm is run on the same input with a mergeable configuration. This allows algorithms to be composed of implicitly reusable algorithms
without publicly sharing intermediate DataSets.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 3965_delegating_graphalgorithm
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2032.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2032
----
commit 615ba45e83461b482a5902811610f26ff2d7a258
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-05-25T10:43:41Z
[FLINK-3965] [gelly] Delegating GraphAlgorithm
A DelegatingGraphAlgorithm wraps a GraphAlgorithm result with a
delegating proxy object. The delegated object can be replaced when the
same algorithm is run on the same input with a mergeable configuration.
This allows algorithms to be composed of implicitly reusable algorithms
without publicly sharing intermediate DataSets.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on the issue:
https://github.com/apache/flink/pull/2032
Hi,
I don't completely get the reasoning either, but it sounds interesting indeed!
@greghogan, could you maybe clarify the following for me:
- What do you mean by "mergeable" configuration? Can you give an example?
- Is the `DelegatingGraphAlgorithm` intended for internal Gelly use, i.e. for the library implementations or is this something that we'd expose to Gelly users as another abstraction?
I know that I haven't been very active during the past weeks, but I see that we now have merged also a `GraphAnalytic` and `AbstractGraphAnalytic` in addition to the existing `GraphAlgorithm` and we also have "Graph Algorithms" under `org.apache.flink.graph.asm` according to the docs. To be honest, I don't completely understand the differences among all of these abstractions and I'm a bit worried we might confuse users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2032
Hi @vasia,
Configurations can be merged if there exists a configuration which satisfies all configuration constraints. It is left to the algorithm to decide whether and how to merged the values.
I see the library algorithms as exemplars for and inspiration to Gelly users.
A `GraphAnalytic` encapsulates accessing computed values via accumulators after execution. Do you think this functionality could be added to `GraphAlgorithm`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2032
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2032
@StephanEwen do you think this is a good and proper strategy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2032
I see no problem with merging this, from a code perspective.
I do not completely get all the mechanics and reasoning behind this, so would be great if @vasia could comment on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/2032#issuecomment-221835760
Sounds very interesting. Can you explain a bit more (at an example maybe) what this does?
I think I lack the background to understand this simply from the description here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/2032#issuecomment-221993882
There is more discussion in the ticket description, but for Gelly the idea is to keep algorithms small and discrete yet not duplicate computation.
My first take on this merely cached algorithm outputs and if the configurations were "equal" would use the prior result.
This second take is able to merge configurations which is much more powerful. We can do this because `DataSet`s are lazily evaluated and we can replace the old `DataSet` when we want to change how we generate the result. We "replace" the `DataSet` by actually wrapping it in a proxy class for which the `MethodHandler` always defers to the replaceable `DataSet`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2032: [FLINK-3965] [gelly] Delegating GraphAlgorithm
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on the issue:
https://github.com/apache/flink/pull/2032
I've had an offline discussion with @greghogan about this. We won't be advertising `DelegatingGraphAlgorithm` as a user-facing feature. +1 from me!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---