You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by vtslab <gi...@git.apache.org> on 2015/12/04 21:12:39 UTC
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
GitHub user vtslab opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/170
Additions to hadoop-gremlin conf files
Hi Marko,
Following your request in:
https://groups.google.com/forum/#!topic/gremlin-users/aE1plgCMhTs
please find the pull request asked for.
I put it in a branch as I feel there is more to it than this. I think gremlin.sh should accept SPARK_HOME and YARN_HOME env variables (in addition to HADOOP_GREMLIN_LIBS) and set the CLASS_PATH accordingly. Also the user documentation could mention the spark-yarn option. I could prepare an additional pull request for that.
Regards, Marc
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vtslab/incubator-tinkerpop spark-yarn-guiding
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/170.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #170
----
commit d971a9002a1a2af7b574fccff2d69052669e9892
Author: MeteorMarc <vt...@xs4all.nl>
Date: 2015-12-04T19:54:12Z
Additions to hadoop-gremlin conf files
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#issuecomment-163653162
Thanks for submitting this - it will be reviewed/merged closer to the release of 3.1.1 as there is some other review of the hadoop configs that package with the distribution that needs to be done first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#issuecomment-178000619
Yes. This is more for @dkuppitz as will get getting GiraphGraphComputer working on the Blades and once he gets it solid, he will be able to go through each recommended property and decide whether what we need and not need.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#issuecomment-221426043
@okram @dkuppitz is there anything we want to do with this one? work it into documentation somehow?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#issuecomment-177971194
@okram are we now looking at this one for 3.1.2 at this point?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#issuecomment-221479227
One day I tried to use `yarn-client` and got all kind of weird errors. I will try again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: Additions to hadoop-gremlin conf...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on a diff in the pull request:
https://github.com/apache/incubator-tinkerpop/pull/170#discussion_r64518418
--- Diff: hadoop-gremlin/conf/hadoop-graphson.properties ---
@@ -28,14 +28,43 @@ gremlin.hadoop.jarsInDistributedCache=true
# the vertex program to execute
gremlin.vertexProgram=org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram
-####################################
-# SparkGraphComputer Configuration #
-####################################
+############################################
+# SparkGraphComputer Configuration (Local) #
+############################################
spark.master=local[4]
-#####################################
-# GiraphGraphComputer Configuration #
-#####################################
+############################################
+# SparkGraphComputer Configuration (Yarn) #
+############################################
+#spark.master=yarn-client
+# app name in the cluster's history server
+#spark.app.name=TinkerPop
+# faster startup if many cluster users ask for the default 4040 port
+#spark.ui.port=4050
+# needed if the default JAVA_HOME on the cluster is not jdk1.8
+#spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/jre
+#spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/jre
+# below point to your cluster's worker node configs
+#spark.executorEnv.SPARK_CONF_DIR=/etc/spark/conf
+#spark.executorEnv.HADOOP_CONF_DIR=/etc/hadoop/conf
+# needed for Hortonworks Data Platform
+#spark.executor.extraJavaOptions=-Dhdp.version=2.3.2.0-2950
+# optional parameters for tuning your TinkerPop application
+#spark.executor.instances=10
+#spark.executor.memory=4g
+#spark.executor.cores=2
+# needed on a secure cluster to allow for multiple computations
+# on a single Kerberos ticket
+#gremlin.spark.persistContext=true
+
+# Below copy all properties from your spark-client's spark-defaults.conf
--- End diff --
Is that necessary? Recommended? The purpose of `*-defaults.conf` files is to set up defaults which you don't want to specify somewhere else, no?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---