You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ankurdave <gi...@git.apache.org> on 2014/07/05 12:11:38 UTC
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
GitHub user ankurdave opened a pull request:
https://github.com/apache/spark/pull/1307
[SPARK-2374] Path pattern matching for GraphX
Based on a [request](http://apache-spark-user-list.1001560.n3.nabble.com/Graphx-traversal-and-merge-interesting-edges-td8788.html) on the mailing list, I wrote a simple implementation of path pattern matching for GraphX. It accepts patterns in the form of sequences of edge matchers, then iteratively propagates partial pattern matches to find all matching paths in the graph.
Though this is only an initial implementation and there are many opportunities for optimization, having this algorithm in the library expands the scope of GraphX beyond ML-like algorithms into graph traversal.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankurdave/spark PatternMatching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1307.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1307
----
commit bc546a22066365122b2cbf4402946b88ee81de7b
Author: Ankur Dave <an...@gmail.com>
Date: 2014-07-05T09:46:34Z
Add graphx.lib.PatternMatching
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-65892530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24211/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48082948
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by ankurdave <gi...@git.apache.org>.
Github user ankurdave commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-81486174
Echoing my earlier comment, this is too inefficient to be part of the standard distribution, so I'm closing this for now. (I don't remember why I reopened it...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083785
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by ankurdave <gi...@git.apache.org>.
Github user ankurdave closed the pull request at:
https://github.com/apache/spark/pull/1307
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083048
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-65890780
[Test build #24211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24211/consoleFull) for PR 1307 at commit [`9332a49`](https://github.com/apache/spark/commit/9332a4927a065e2e5217a4256a5bc12a127ca97b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083786
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16354/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083043
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by ankurdave <gi...@git.apache.org>.
GitHub user ankurdave reopened a pull request:
https://github.com/apache/spark/pull/1307
[SPARK-2374] Path pattern matching for GraphX
Based on a [request](http://apache-spark-user-list.1001560.n3.nabble.com/Graphx-traversal-and-merge-interesting-edges-td8788.html) on the mailing list, I wrote a simple implementation of path pattern matching for GraphX. It accepts patterns in the form of sequences of edge matchers, then iteratively propagates partial pattern matches to find all matching paths in the graph.
Though this is only an initial implementation and there are many opportunities for optimization, having this algorithm in the library expands the scope of GraphX beyond ML-like algorithms into graph traversal.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankurdave/spark PatternMatching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1307.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1307
----
commit bc546a22066365122b2cbf4402946b88ee81de7b
Author: Ankur Dave <an...@gmail.com>
Date: 2014-07-05T09:46:34Z
Add graphx.lib.PatternMatching
commit 9332a4927a065e2e5217a4256a5bc12a127ca97b
Author: Ankur Dave <an...@gmail.com>
Date: 2014-07-05T10:16:20Z
Fix comment error
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48082949
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by eranation <gi...@git.apache.org>.
Github user eranation commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-63115547
Are there any plans to approve this PR? is there any alternative solution in development for pattern matching?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by ankurdave <gi...@git.apache.org>.
Github user ankurdave commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-63451454
@eranation I'd like to resubmit it at some point, but I'm worried it's too inefficient at the moment to put into the main distribution. It would be great to explore any indexing strategies that could avoid the brute force flooding approach it currently uses.
Still, if you need pattern matching and you have enough memory, it might be worth trying it as-is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083120
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-65892529
[Test build #24211 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24211/consoleFull) for PR 1307 at commit [`9332a49`](https://github.com/apache/spark/commit/9332a4927a065e2e5217a4256a5bc12a127ca97b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class EdgePattern[ED](attr: ED, matchDstFirst: Boolean = false) extends Serializable `
* `case class EdgeMatch[ED](srcId: VertexId, dstId: VertexId, attr: ED) extends Serializable `
* `case class Match[ED](path: List[EdgeMatch[ED]]) extends Serializable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by ankurdave <gi...@git.apache.org>.
Github user ankurdave closed the pull request at:
https://github.com/apache/spark/pull/1307
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1307#issuecomment-48083121
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16353/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---