You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Xi Shen <da...@gmail.com> on 2016/10/09 06:36:56 UTC
How to connect to a remote YARN cluster?
Hi,
I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html,
and set up SPARK_HOME, HADOOP_CONF_DIR.
My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web.
After I start Zeppelin, I went to the interpreter setting page and changed
Spark interpreter settings as follow:
master: yarn
deploy-mode: client
Then, I created a new notebook and executed:
%spark
spark.version
The block never finishes...no error either.
In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which
I think is the cause of my problem*
INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2}
Logging.scala[logInfo]:54) - Added JAR
*file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar*
at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with
timestamp 1475994517073
INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2}
Logging.scala[logInfo]:54) - Created default pool default, schedulingMode:
FIFO, minShare: 0, weight: 1
INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2}
RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at
/0.0.0.0:8032 <http://0.0.0.0:8032>*
Looks like Zeppelin is not using my Spark binary, and is not using my
hadoop configuration.
What did I miss?
--
Thanks,
David S.
Re: How to connect to a remote YARN cluster?
Posted by Xi Shen <da...@gmail.com>.
Yes, my configuration files are at /opt/hadoop/conf...but the configuration
was not correct. It's working now.
On Sun, Oct 9, 2016 at 3:54 PM Jeff Zhang <zj...@gmail.com> wrote:
> There're 2 processes, the first one is embedded mode. The second is the
> normal spark launch mode. I am not sure which process you are using, can
> you kill these processes and rerun again ?
>
> And are you sure you put hadoop conf files under /opt/hadoop/conf/ ?
>
> On Sun, Oct 9, 2016 at 3:14 PM, Xi Shen <da...@gmail.com> wrote:
>
>
> ps aux | grep RemoteInterpreterServer
>
> root 71 0.1 3.3 3528164 133668 ? Sl 06:21 0:04
> /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Dfile.encoding=UTF-8
> -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
> -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-sh--cad975db8a21.log
> -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp
> ::/opt/zeppelin-0.6.1-bin-all/interpreter/sh/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar
> org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer *45877
> root 346 0.7 13.7 3612088 555092 ? Sl 06:28 0:18
> /usr/lib/jvm/java-8-openjdk-amd64//bin/java -cp
> /opt/zeppelin-0.6.1-bin-all/interpreter/spark/*:/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar:/opt/spark/conf/:/opt/spark/jars/*:/opt/hadoop/conf/
> -Xmx1g -Dfile.encoding=UTF-8
> -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
> -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log
> org.apache.spark.deploy.SparkSubmit --conf
> spark.driver.extraClassPath=::/opt/zeppelin-0.6.1-bin-all/interpreter/spark/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar
> --conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8
> -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
> -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log
> --class org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer */opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar
> 44320
>
> I started my Zeppelin instance with
>
> /opt/zeppelin-0.6.1-bin-all/bin/zeppelin.sh --conf
> /opt/zeppelin-0.6.1-bin-all/conf
>
> On Sun, Oct 9, 2016 at 2:54 PM Jeff Zhang <zj...@gmail.com> wrote:
>
> Could you check the classpath of the interpreter process ? I suspect you
> didn't export HADOOP_CONF_DIR correctly.
>
> Run the following command:
>
> ps aux | grep RemoteInterpreterServer
>
> On Sun, Oct 9, 2016 at 2:36 PM, Xi Shen <da...@gmail.com> wrote:
>
> Hi,
>
> I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html,
> and set up SPARK_HOME, HADOOP_CONF_DIR.
>
> My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web.
>
> After I start Zeppelin, I went to the interpreter setting page and changed
> Spark interpreter settings as follow:
>
> master: yarn
> deploy-mode: client
>
> Then, I created a new notebook and executed:
>
> %spark
>
> spark.version
>
> The block never finishes...no error either.
>
> In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which
> I think is the cause of my problem*
>
> INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Added JAR
> *file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar*
> at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with
> timestamp 1475994517073
> INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Created default pool default, schedulingMode:
> FIFO, minShare: 0, weight: 1
> INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2}
> RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at
> /0.0.0.0:8032 <http://0.0.0.0:8032>*
>
> Looks like Zeppelin is not using my Spark binary, and is not using my
> hadoop configuration.
>
> What did I miss?
>
> --
>
>
> Thanks,
> David S.
>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
> --
>
>
> Thanks,
> David S.
>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Thanks,
David S.
Re: How to connect to a remote YARN cluster?
Posted by Jeff Zhang <zj...@gmail.com>.
There're 2 processes, the first one is embedded mode. The second is the
normal spark launch mode. I am not sure which process you are using, can
you kill these processes and rerun again ?
And are you sure you put hadoop conf files under /opt/hadoop/conf/ ?
On Sun, Oct 9, 2016 at 3:14 PM, Xi Shen <da...@gmail.com> wrote:
>
> ps aux | grep RemoteInterpreterServer
>
> root 71 0.1 3.3 3528164 133668 ? Sl 06:21 0:04
> /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Dfile.encoding=UTF-8
> -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
> -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/
> zeppelin-interpreter-sh--cad975db8a21.log -Xms1024m -Xmx1024m
> -XX:MaxPermSize=512m -cp ::/opt/zeppelin-0.6.1-bin-all/
> interpreter/sh/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar
> org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer *45877
> root 346 0.7 13.7 3612088 555092 ? Sl 06:28 0:18
> /usr/lib/jvm/java-8-openjdk-amd64//bin/java -cp
> /opt/zeppelin-0.6.1-bin-all/interpreter/spark/*:/opt/
> zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.
> jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/
> zeppelin-spark_2.11-0.6.1.jar:/opt/spark/conf/:/opt/spark/jars/*:/opt/hadoop/conf/
> -Xmx1g -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///
> opt/zeppelin-0.6.1-bin-all/conf/log4j.properties -Dzeppelin.log.file=/opt/
> zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log
> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=::
> /opt/zeppelin-0.6.1-bin-all/interpreter/spark/*::/opt/
> zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.
> jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar
> --conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8
> -Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
> -Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/
> zeppelin-interpreter-spark--cad975db8a21.log --class org.apache.zeppelin.
> interpreter.remote.*RemoteInterpreterServer */opt/zeppelin-0.6.1-bin-all/
> interpreter/spark/zeppelin-spark_2.11-0.6.1.jar 44320
>
> I started my Zeppelin instance with
>
> /opt/zeppelin-0.6.1-bin-all/bin/zeppelin.sh --conf
> /opt/zeppelin-0.6.1-bin-all/conf
>
> On Sun, Oct 9, 2016 at 2:54 PM Jeff Zhang <zj...@gmail.com> wrote:
>
>> Could you check the classpath of the interpreter process ? I suspect you
>> didn't export HADOOP_CONF_DIR correctly.
>>
>> Run the following command:
>>
>> ps aux | grep RemoteInterpreterServer
>>
>> On Sun, Oct 9, 2016 at 2:36 PM, Xi Shen <da...@gmail.com> wrote:
>>
>> Hi,
>>
>> I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html,
>> and set up SPARK_HOME, HADOOP_CONF_DIR.
>>
>> My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web.
>>
>> After I start Zeppelin, I went to the interpreter setting page and
>> changed Spark interpreter settings as follow:
>>
>> master: yarn
>> deploy-mode: client
>>
>> Then, I created a new notebook and executed:
>>
>> %spark
>>
>> spark.version
>>
>> The block never finishes...no error either.
>>
>> In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which
>> I think is the cause of my problem*
>>
>> INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2}
>> Logging.scala[logInfo]:54) - Added JAR
>> *file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar*
>> at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with
>> timestamp 1475994517073
>> INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2}
>> Logging.scala[logInfo]:54) - Created default pool default, schedulingMode:
>> FIFO, minShare: 0, weight: 1
>> INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2}
>> RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at
>> /0.0.0.0:8032 <http://0.0.0.0:8032>*
>>
>> Looks like Zeppelin is not using my Spark binary, and is not using my
>> hadoop configuration.
>>
>> What did I miss?
>>
>> --
>>
>>
>> Thanks,
>> David S.
>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
> --
>
>
> Thanks,
> David S.
>
--
Best Regards
Jeff Zhang
Re: How to connect to a remote YARN cluster?
Posted by Xi Shen <da...@gmail.com>.
ps aux | grep RemoteInterpreterServer
root 71 0.1 3.3 3528164 133668 ? Sl 06:21 0:04
/usr/lib/jvm/java-8-openjdk-amd64//bin/java -Dfile.encoding=UTF-8
-Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
-Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-sh--cad975db8a21.log
-Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp
::/opt/zeppelin-0.6.1-bin-all/interpreter/sh/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar
org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer *45877
root 346 0.7 13.7 3612088 555092 ? Sl 06:28 0:18
/usr/lib/jvm/java-8-openjdk-amd64//bin/java -cp
/opt/zeppelin-0.6.1-bin-all/interpreter/spark/*:/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar:/opt/spark/conf/:/opt/spark/jars/*:/opt/hadoop/conf/
-Xmx1g -Dfile.encoding=UTF-8
-Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
-Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log
org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=::/opt/zeppelin-0.6.1-bin-all/interpreter/spark/*::/opt/zeppelin-0.6.1-bin-all/lib/zeppelin-interpreter-0.6.1.jar:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar
--conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8
-Dlog4j.configuration=file:///opt/zeppelin-0.6.1-bin-all/conf/log4j.properties
-Dzeppelin.log.file=/opt/zeppelin-0.6.1-bin-all/logs/zeppelin-interpreter-spark--cad975db8a21.log
--class org.apache.zeppelin.interpreter.remote.*RemoteInterpreterServer
*/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar
44320
I started my Zeppelin instance with
/opt/zeppelin-0.6.1-bin-all/bin/zeppelin.sh --conf
/opt/zeppelin-0.6.1-bin-all/conf
On Sun, Oct 9, 2016 at 2:54 PM Jeff Zhang <zj...@gmail.com> wrote:
> Could you check the classpath of the interpreter process ? I suspect you
> didn't export HADOOP_CONF_DIR correctly.
>
> Run the following command:
>
> ps aux | grep RemoteInterpreterServer
>
> On Sun, Oct 9, 2016 at 2:36 PM, Xi Shen <da...@gmail.com> wrote:
>
> Hi,
>
> I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html,
> and set up SPARK_HOME, HADOOP_CONF_DIR.
>
> My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web.
>
> After I start Zeppelin, I went to the interpreter setting page and changed
> Spark interpreter settings as follow:
>
> master: yarn
> deploy-mode: client
>
> Then, I created a new notebook and executed:
>
> %spark
>
> spark.version
>
> The block never finishes...no error either.
>
> In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which
> I think is the cause of my problem*
>
> INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Added JAR
> *file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar*
> at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with
> timestamp 1475994517073
> INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Created default pool default, schedulingMode:
> FIFO, minShare: 0, weight: 1
> INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2}
> RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at
> /0.0.0.0:8032 <http://0.0.0.0:8032>*
>
> Looks like Zeppelin is not using my Spark binary, and is not using my
> hadoop configuration.
>
> What did I miss?
>
> --
>
>
> Thanks,
> David S.
>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Thanks,
David S.
Re: How to connect to a remote YARN cluster?
Posted by Jeff Zhang <zj...@gmail.com>.
Could you check the classpath of the interpreter process ? I suspect you
didn't export HADOOP_CONF_DIR correctly.
Run the following command:
ps aux | grep RemoteInterpreterServer
On Sun, Oct 9, 2016 at 2:36 PM, Xi Shen <da...@gmail.com> wrote:
> Hi,
>
> I followed http://zeppelin.apache.org/docs/latest/interpreter/spark.html,
> and set up SPARK_HOME, HADOOP_CONF_DIR.
>
> My SPARK build is 2.0. My Zeppelin build is the 0.6.1 binary from the web.
>
> After I start Zeppelin, I went to the interpreter setting page and changed
> Spark interpreter settings as follow:
>
> master: yarn
> deploy-mode: client
>
> Then, I created a new notebook and executed:
>
> %spark
>
> spark.version
>
> The block never finishes...no error either.
>
> In the ./logs/zeppelin-interpreter-spark-*.log, I found the following *which
> I think is the cause of my problem*
>
> INFO [2016-10-09 06:28:37,074] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Added JAR
> *file:/opt/zeppelin-0.6.1-bin-all/interpreter/spark/zeppelin-spark_2.11-0.6.1.jar*
> at spark://172.17.0.3:38775/jars/zeppelin-spark_2.11-0.6.1.jar with
> timestamp 1475994517073
> INFO [2016-10-09 06:28:37,150] ({pool-2-thread-2}
> Logging.scala[logInfo]:54) - Created default pool default, schedulingMode:
> FIFO, minShare: 0, weight: 1
> INFO [2016-10-09 06:28:38,205] ({pool-2-thread-2}
> RMProxy.java[createRMProxy]:98) - *Connecting to ResourceManager at
> /0.0.0.0:8032 <http://0.0.0.0:8032>*
>
> Looks like Zeppelin is not using my Spark binary, and is not using my
> hadoop configuration.
>
> What did I miss?
>
> --
>
>
> Thanks,
> David S.
>
--
Best Regards
Jeff Zhang