You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by RussellSpitzer <gi...@git.apache.org> on 2015/11/03 00:25:43 UTC
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
GitHub user RussellSpitzer opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/132
TINKERPOP-911: Lets SparkGraphComputer Set ThreadLocal Properties
When running in persistent context mode, the configuration for the Spark
Context is always inherited from the origin configuration. This can be
overridden for select properties using setLocalProperty. By enabling
this for SparkGraphComputer a user will be able to take advantage of the
FairScheduler system with spark or add custom Spark JobGroupIDs to jobs in the
same JVM.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/RussellSpitzer/incubator-tinkerpop TINKERPOP-911
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #132
----
commit 7502464109ffe9440274da13c70e3bdc431caf10
Author: Russell Spitzer <ru...@gmail.com>
Date: 2015-11-02T23:22:45Z
TINKERPOP-911: Lets SparkGraphComputer Set ThreadLocal Properties
When running in persistent context mode, the configuration for the Spark
Context is always inherited from the origin configuration. This can be
overridden for select properties using setLocalProperty. By enabling
this for SparkGraphComputer a user will be able to take advantage of the
FairScheduler system with spark or add custom Spark JobGroupIDs to jobs in the
same JVM.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153461629
@RussellSpitzer if your docs don't contain code examples you can typically get away with:
```text
bin/process-docs.sh --dryRun
```
which just runs the asciidoc processor. To see code examples generated too with output you gotta have all the other infrastructure running (i.e. hadoop, etc.). In your case the `--dryRun` would seem sufficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by RussellSpitzer <gi...@git.apache.org>.
Github user RussellSpitzer commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153799691
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache TinkerPop ................................... SUCCESS [ 3.445 s]
[INFO] Apache TinkerPop :: Gremlin Shaded ................. SUCCESS [ 1.633 s]
[INFO] Apache TinkerPop :: Gremlin Core ................... SUCCESS [ 25.242 s]
[INFO] Apache TinkerPop :: Gremlin Test ................... SUCCESS [ 9.935 s]
[INFO] Apache TinkerPop :: Gremlin Groovy ................. SUCCESS [ 36.015 s]
[INFO] Apache TinkerPop :: Gremlin Groovy Test ............ SUCCESS [ 4.602 s]
[INFO] Apache TinkerPop :: TinkerGraph Gremlin ............ SUCCESS [01:32 min]
[INFO] Apache TinkerPop :: Hadoop Gremlin ................. SUCCESS [04:03 min]
[INFO] Apache TinkerPop :: Spark Gremlin .................. SUCCESS [ 35.039 s]
[INFO] Apache TinkerPop :: Giraph Gremlin ................. SUCCESS [ 4.346 s]
[INFO] Apache TinkerPop :: Neo4j Gremlin .................. SUCCESS [ 2.158 s]
[INFO] Apache TinkerPop :: Gremlin Driver ................. SUCCESS [ 6.774 s]
[INFO] Apache TinkerPop :: Gremlin Server ................. SUCCESS [ 3.643 s]
[INFO] Apache TinkerPop :: Gremlin Console ................ SUCCESS [ 11.334 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:00 min
[INFO] Finished at: 2015-11-04T09:18:53-08:00
[INFO] Final Memory: 124M/1304M```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by PommeVerte <gi...@git.apache.org>.
Github user PommeVerte commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153672244
I couldn't really test this as `mvn clean install` fails for me even on `master` with : `InputRDDTest.shouldReadFromArbitraryRDD:51 ยป IllegalState org.apache.spark.Spa...`
(`spark-gremlin` project)
Am I the only one with this issue?
That failure aside, the docs build correctly and the spark integration tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tinkerpop/pull/132
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153425920
Looks good to me, now that the docs are done.
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by PommeVerte <gi...@git.apache.org>.
Github user PommeVerte commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153817872
I finally managed to build this... It's a bit of a mystery as to why it works now, but that's good for me.
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by PommeVerte <gi...@git.apache.org>.
Github user PommeVerte commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153295330
Hey, should there be a mention of this somewhere in the documentation or are javadocs enough in this instance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by RussellSpitzer <gi...@git.apache.org>.
Github user RussellSpitzer commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153423294
Added in some documentation for this and an explicit list of allowable properties, the ascii doc looks ok in my IDEA preview, is there a way to build the docs and view them locally? I'd hate to break them again @okram :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-911: Lets SparkGraphCo...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/132#issuecomment-153366664
This looks good to me. Test case seems legit (though I would like to simply this style which is also seen in `PersistedInputOutputRDDTest`. As @PommeVerte said -- do we need to mention this in docs? If so, please add to PR along with update to CHANGELOG.
VOTE +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---