You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Benoit Hanotte <be...@gmail.com> on 2016/11/01 20:07:16 UTC
Zeppelin in local computer using yarn on distant cluster
Hello,
I'd like to use zeppelin on my local computer and use it to run spark
executors on a distant yarn cluster since I can't easily install zeppelin
on the cluster gateway.
I installed the correct hadoop version (2.6), and compiled zeppelin (from
the master branch) as following:
*mvn clean package -DskipTests -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.5.0
-Pyarn -Pspark-2.0 -Pscala-2.11*
I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
to the resource manager of the cluster (I copied the value from the config
file on the cluster) but when I start a spark command it still tries to
connect to 0.0.0.0:8032 as one can see in the logs:
*INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
Client.java[handleConnectionFailure]:862) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)*
Am I missing something something? Is there any additional parameters to set?
Thanks!
Benoit
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Benoit Hanotte <be...@gmail.com>.
I am pointing to the dirs on my local machine, what I want is simply for my
jobs to be submitted to the distant yarn cluster
Thanks
On Wed, Nov 2, 2016 at 4:00 PM, Abhi Basu <90...@gmail.com> wrote:
> I am assuming you are pointing to hadoop/spark on remote host, right? Can
> you not point hadoop conf and spark dirs to remote machine? Not sure if
> this works, just suggesting, others may have tried.
>
> On Wed, Nov 2, 2016 at 9:58 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
>
>> Hello.
>> You don't need to install hadoop in your machine but you need a proper
>> version of spark[0] to use spark-submit.
>> and then you can set[1] the SPARK_HOME where the spark exists and
>> HADOOP_CONF_DIR, master as yarn-client your spark interpreter in the
>> interpreter menu.
>>
>> [0]
>> http://spark.apache.org/downloads.html
>> [1]
>> http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/
>> spark.html#1-export-spark_home
>> http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/spark
>> _cluster_mode.html#4-configure-spark-interpreter-in-zeppelin
>>
>> Hope this helps.
>>
>> 2016-11-02 19:06 GMT+09:00 Benoit Hanotte <be...@gmail.com>:
>>
>>> I have only set HADOOP_CONF_DIR as following (my hadoop conf files are
>>> in /usr/local/lib/hadoop/etc/hadoop/, eg /usr/local/lib/hadoop/etc/hado
>>> op/yarn-site.xml):
>>>
>>> #!/bin/bash
>>> #
>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>> # contributor license agreements. See the NOTICE file distributed
>>> with
>>> # this work for additional information regarding copyright ownership.
>>> # The ASF licenses this file to You under the Apache License,
>>> Version 2.0
>>> # (the "License"); you may not use this file except in compliance
>>> with
>>> # the License. You may obtain a copy of the License at
>>> #
>>> # http://www.apache.org/licenses/LICENSE-2.0
>>> #
>>> # Unless required by applicable law or agreed to in writing, software
>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> implied.
>>> # See the License for the specific language governing permissions and
>>> # limitations under the License.
>>> #
>>>
>>> # export JAVA_HOME=
>>> # export MASTER= # Spark master url. eg.
>>> spark://master_addr:7077. Leave empty if you want to use local mode.
>>> # export ZEPPELIN_JAVA_OPTS # Additional jvm options. for
>>> example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g
>>> -Dspark.cores.max=16"
>>> # export ZEPPELIN_MEM # Zeppelin jvm mem options
>>> Default -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
>>> # export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm
>>> mem options. Default -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
>>> # export ZEPPELIN_INTP_JAVA_OPTS # zeppelin interpreter process jvm
>>> options.
>>> # export ZEPPELIN_SSL_PORT # ssl port (used when ssl
>>> environment variable is set to true)
>>>
>>> # export ZEPPELIN_LOG_DIR # Where log files are stored.
>>> PWD by default.
>>> # export ZEPPELIN_PID_DIR # The pid files are stored.
>>> ${ZEPPELIN_HOME}/run by default.
>>> # export ZEPPELIN_WAR_TEMPDIR # The location of jetty temporary
>>> directory.
>>> # export ZEPPELIN_NOTEBOOK_DIR # Where notebook saved
>>> # export ZEPPELIN_NOTEBOOK_HOMESCREEN # Id of notebook to be
>>> displayed in homescreen. ex) 2A94M5J1Z
>>> # export ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE # hide homescreen
>>> notebook from list when this value set to "true". default "false"
>>> # export ZEPPELIN_NOTEBOOK_S3_BUCKET # Bucket where notebook
>>> saved
>>> # export ZEPPELIN_NOTEBOOK_S3_ENDPOINT # Endpoint of the bucket
>>> # export ZEPPELIN_NOTEBOOK_S3_USER # User in bucket where
>>> notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json
>>> # export ZEPPELIN_IDENT_STRING # A string representing this
>>> instance of zeppelin. $USER by default.
>>> # export ZEPPELIN_NICENESS # The scheduling priority for
>>> daemons. Defaults to 0.
>>> # export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository
>>> for interpreter's additional dependency loading
>>> # export ZEPPELIN_NOTEBOOK_STORAGE # Refers to pluggable notebook
>>> storage class, can have two classes simultaneously with a sync between them
>>> (e.g. local and remote).
>>> # export ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC # If there are multiple
>>> notebook storages, should we treat the first one as the only source of
>>> truth?
>>>
>>> #### Spark interpreter configuration ####
>>>
>>> ## Use provided spark installation ##
>>> ## defining SPARK_HOME makes Zeppelin run spark interpreter process
>>> using spark-submit
>>> ##
>>> # export SPARK_HOME # (required) When it
>>> is defined, load it instead of Zeppelin embedded Spark libraries
>>> # export SPARK_SUBMIT_OPTIONS # (optional) extra
>>> options to pass to spark submit. eg) "--driver-memory 512M
>>> --executor-memory 1G".
>>> # export SPARK_APP_NAME # (optional) The
>>> name of spark application.
>>>
>>> ## Use embedded spark binaries ##
>>> ## without SPARK_HOME defined, Zeppelin still able to run spark
>>> interpreter process using embedded spark binaries.
>>> ## however, it is not encouraged when you can define SPARK_HOME
>>> ##
>>> # Options read in YARN client mode
>>> export HADOOP_CONF_DIR = /usr/local/lib/hadoop/etc/hadoop/ #
>>> yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
>>> # Pyspark (supported with Spark 1.2.1 and above)
>>> # To configure pyspark, you need to set spark distribution's path to
>>> 'spark.home' property in Interpreter setting screen in Zeppelin GUI
>>> # export PYSPARK_PYTHON # path to the python command.
>>> must be the same path on the driver(Zeppelin) and all workers.
>>> # export PYTHONPATH
>>>
>>> ## Spark interpreter options ##
>>> ##
>>> # export ZEPPELIN_SPARK_USEHIVECONTEXT # Use HiveContext instead of
>>> SQLContext if set true. true by default.
>>> # export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL
>>> concurrently if set true. false by default.
>>> # export ZEPPELIN_SPARK_IMPORTIMPLICIT # Import implicits, UDF
>>> collection, and sql if set true. true by default.
>>> # export ZEPPELIN_SPARK_MAXRESULT # Max number of Spark SQL
>>> result to display. 1000 by default.
>>> # export ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE # Size in
>>> characters of the maximum text message to be received by websocket.
>>> Defaults to 1024000
>>>
>>>
>>> #### HBase interpreter configuration ####
>>>
>>> ## To connect to HBase running on a cluster, either HBASE_HOME or
>>> HBASE_CONF_DIR must be set
>>>
>>> # export HBASE_HOME= # (require) Under which
>>> HBase scripts and configuration should be
>>> # export HBASE_CONF_DIR= # (optional) Alternatively,
>>> configuration directory can be set to point to the directory that has
>>> hbase-site.xml
>>>
>>> #### ZeppelinHub connection configuration ####
>>> # export ZEPPELINHUB_API_ADDRESS # Refers to the address of the
>>> ZeppelinHub service in use
>>> # export ZEPPELINHUB_API_TOKEN # Refers to the Zeppelin instance
>>> token of the user
>>> # export ZEPPELINHUB_USER_KEY # Optional, when using Zeppelin with
>>> authentication.
>>>
>>>
>>>
>>> I also tried simply /usr/local/lib/hadoop and I also create a conf
>>> directory within /usr/local/lib/hadoop/etc/hadoop and placed
>>> yarn-site.xml within this folder
>>>
>>> Thanks
>>>
>>> On Wed, Nov 2, 2016 at 10:06 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>>> Could you share your zeppelin-env.sh ?
>>>> 2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <be...@gmail.com>님이 작성:
>>>>
>>>>> Thanks for your reply,
>>>>> I have tried setting it within zeppelin-env.sh but it doesn't work any
>>>>> better.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>> Hello.
>>>>> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/
>>>>> in the conf/zeppelin-env.sh.
>>>>> Thanks.
>>>>> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'd like to use zeppelin on my local computer and use it to run spark
>>>>> executors on a distant yarn cluster since I can't easily install zeppelin
>>>>> on the cluster gateway.
>>>>>
>>>>> I installed the correct hadoop version (2.6), and compiled zeppelin
>>>>> (from the master branch) as following:
>>>>>
>>>>> *mvn clean package -DskipTests -Phadoop-2.6
>>>>> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>>>>>
>>>>> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
>>>>> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where
>>>>> the conf files such as yarn-site.xml are). I set
>>>>> yarn.resourcemanager.hostname to the resource manager of the cluster (I
>>>>> copied the value from the config file on the cluster) but when I start a
>>>>> spark command it still tries to connect to 0.0.0.0:8032 as one can
>>>>> see in the logs:
>>>>>
>>>>> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
>>>>> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
>>>>> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
>>>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>>>> sleepTime=1000 MILLISECONDS)*
>>>>>
>>>>> Am I missing something something? Is there any additional parameters
>>>>> to set?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Benoit
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>
>
> --
> Abhi Basu
>
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Abhi Basu <90...@gmail.com>.
I am assuming you are pointing to hadoop/spark on remote host, right? Can
you not point hadoop conf and spark dirs to remote machine? Not sure if
this works, just suggesting, others may have tried.
On Wed, Nov 2, 2016 at 9:58 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Hello.
> You don't need to install hadoop in your machine but you need a proper
> version of spark[0] to use spark-submit.
> and then you can set[1] the SPARK_HOME where the spark exists and
> HADOOP_CONF_DIR, master as yarn-client your spark interpreter in the
> interpreter menu.
>
> [0]
> http://spark.apache.org/downloads.html
> [1]
> http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/spark.html#1-
> export-spark_home
> http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/
> spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin
>
> Hope this helps.
>
> 2016-11-02 19:06 GMT+09:00 Benoit Hanotte <be...@gmail.com>:
>
>> I have only set HADOOP_CONF_DIR as following (my hadoop conf files are in
>> /usr/local/lib/hadoop/etc/hadoop/, eg /usr/local/lib/hadoop/etc/hado
>> op/yarn-site.xml):
>>
>> #!/bin/bash
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements. See the NOTICE file distributed
>> with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version
>> 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>>
>> # export JAVA_HOME=
>> # export MASTER= # Spark master url. eg.
>> spark://master_addr:7077. Leave empty if you want to use local mode.
>> # export ZEPPELIN_JAVA_OPTS # Additional jvm options. for
>> example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g
>> -Dspark.cores.max=16"
>> # export ZEPPELIN_MEM # Zeppelin jvm mem options Default
>> -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
>> # export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm
>> mem options. Default -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
>> # export ZEPPELIN_INTP_JAVA_OPTS # zeppelin interpreter process jvm
>> options.
>> # export ZEPPELIN_SSL_PORT # ssl port (used when ssl
>> environment variable is set to true)
>>
>> # export ZEPPELIN_LOG_DIR # Where log files are stored. PWD
>> by default.
>> # export ZEPPELIN_PID_DIR # The pid files are stored.
>> ${ZEPPELIN_HOME}/run by default.
>> # export ZEPPELIN_WAR_TEMPDIR # The location of jetty temporary
>> directory.
>> # export ZEPPELIN_NOTEBOOK_DIR # Where notebook saved
>> # export ZEPPELIN_NOTEBOOK_HOMESCREEN # Id of notebook to be
>> displayed in homescreen. ex) 2A94M5J1Z
>> # export ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE # hide homescreen
>> notebook from list when this value set to "true". default "false"
>> # export ZEPPELIN_NOTEBOOK_S3_BUCKET # Bucket where notebook
>> saved
>> # export ZEPPELIN_NOTEBOOK_S3_ENDPOINT # Endpoint of the bucket
>> # export ZEPPELIN_NOTEBOOK_S3_USER # User in bucket where
>> notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json
>> # export ZEPPELIN_IDENT_STRING # A string representing this
>> instance of zeppelin. $USER by default.
>> # export ZEPPELIN_NICENESS # The scheduling priority for
>> daemons. Defaults to 0.
>> # export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository
>> for interpreter's additional dependency loading
>> # export ZEPPELIN_NOTEBOOK_STORAGE # Refers to pluggable notebook
>> storage class, can have two classes simultaneously with a sync between them
>> (e.g. local and remote).
>> # export ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC # If there are multiple
>> notebook storages, should we treat the first one as the only source of
>> truth?
>>
>> #### Spark interpreter configuration ####
>>
>> ## Use provided spark installation ##
>> ## defining SPARK_HOME makes Zeppelin run spark interpreter process
>> using spark-submit
>> ##
>> # export SPARK_HOME # (required) When it
>> is defined, load it instead of Zeppelin embedded Spark libraries
>> # export SPARK_SUBMIT_OPTIONS # (optional) extra
>> options to pass to spark submit. eg) "--driver-memory 512M
>> --executor-memory 1G".
>> # export SPARK_APP_NAME # (optional) The name
>> of spark application.
>>
>> ## Use embedded spark binaries ##
>> ## without SPARK_HOME defined, Zeppelin still able to run spark
>> interpreter process using embedded spark binaries.
>> ## however, it is not encouraged when you can define SPARK_HOME
>> ##
>> # Options read in YARN client mode
>> export HADOOP_CONF_DIR = /usr/local/lib/hadoop/etc/hadoop/ #
>> yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
>> # Pyspark (supported with Spark 1.2.1 and above)
>> # To configure pyspark, you need to set spark distribution's path to
>> 'spark.home' property in Interpreter setting screen in Zeppelin GUI
>> # export PYSPARK_PYTHON # path to the python command. must
>> be the same path on the driver(Zeppelin) and all workers.
>> # export PYTHONPATH
>>
>> ## Spark interpreter options ##
>> ##
>> # export ZEPPELIN_SPARK_USEHIVECONTEXT # Use HiveContext instead of
>> SQLContext if set true. true by default.
>> # export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL
>> concurrently if set true. false by default.
>> # export ZEPPELIN_SPARK_IMPORTIMPLICIT # Import implicits, UDF
>> collection, and sql if set true. true by default.
>> # export ZEPPELIN_SPARK_MAXRESULT # Max number of Spark SQL
>> result to display. 1000 by default.
>> # export ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE # Size in
>> characters of the maximum text message to be received by websocket.
>> Defaults to 1024000
>>
>>
>> #### HBase interpreter configuration ####
>>
>> ## To connect to HBase running on a cluster, either HBASE_HOME or
>> HBASE_CONF_DIR must be set
>>
>> # export HBASE_HOME= # (require) Under which HBase
>> scripts and configuration should be
>> # export HBASE_CONF_DIR= # (optional) Alternatively,
>> configuration directory can be set to point to the directory that has
>> hbase-site.xml
>>
>> #### ZeppelinHub connection configuration ####
>> # export ZEPPELINHUB_API_ADDRESS # Refers to the address of the
>> ZeppelinHub service in use
>> # export ZEPPELINHUB_API_TOKEN # Refers to the Zeppelin instance
>> token of the user
>> # export ZEPPELINHUB_USER_KEY # Optional, when using Zeppelin with
>> authentication.
>>
>>
>>
>> I also tried simply /usr/local/lib/hadoop and I also create a conf
>> directory within /usr/local/lib/hadoop/etc/hadoop and placed
>> yarn-site.xml within this folder
>>
>> Thanks
>>
>> On Wed, Nov 2, 2016 at 10:06 AM, Hyung Sung Shim <hs...@nflabs.com>
>> wrote:
>>
>>> Could you share your zeppelin-env.sh ?
>>> 2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <be...@gmail.com>님이 작성:
>>>
>>>> Thanks for your reply,
>>>> I have tried setting it within zeppelin-env.sh but it doesn't work any
>>>> better.
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>> wrote:
>>>>
>>>> Hello.
>>>> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/
>>>> in the conf/zeppelin-env.sh.
>>>> Thanks.
>>>> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>>>>
>>>> Hello,
>>>>
>>>> I'd like to use zeppelin on my local computer and use it to run spark
>>>> executors on a distant yarn cluster since I can't easily install zeppelin
>>>> on the cluster gateway.
>>>>
>>>> I installed the correct hadoop version (2.6), and compiled zeppelin
>>>> (from the master branch) as following:
>>>>
>>>> *mvn clean package -DskipTests -Phadoop-2.6
>>>> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>>>>
>>>> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
>>>> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where
>>>> the conf files such as yarn-site.xml are). I set
>>>> yarn.resourcemanager.hostname to the resource manager of the cluster (I
>>>> copied the value from the config file on the cluster) but when I start a
>>>> spark command it still tries to connect to 0.0.0.0:8032 as one can see
>>>> in the logs:
>>>>
>>>> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
>>>> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
>>>> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
>>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>>> sleepTime=1000 MILLISECONDS)*
>>>>
>>>> Am I missing something something? Is there any additional parameters to
>>>> set?
>>>>
>>>> Thanks!
>>>>
>>>> Benoit
>>>>
>>>>
>>>>
>>>>
>>
>
--
Abhi Basu
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Hello.
You don't need to install hadoop in your machine but you need a proper
version of spark[0] to use spark-submit.
and then you can set[1] the SPARK_HOME where the spark exists and
HADOOP_CONF_DIR, master as yarn-client your spark interpreter in the
interpreter menu.
[0]
http://spark.apache.org/downloads.html
[1]
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/spark.html#1-export-spark_home
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin
Hope this helps.
2016-11-02 19:06 GMT+09:00 Benoit Hanotte <be...@gmail.com>:
> I have only set HADOOP_CONF_DIR as following (my hadoop conf files are in
> /usr/local/lib/hadoop/etc/hadoop/, eg /usr/local/lib/hadoop/etc/
> hadoop/yarn-site.xml):
>
> #!/bin/bash
> #
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version
> 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
>
> # export JAVA_HOME=
> # export MASTER= # Spark master url. eg.
> spark://master_addr:7077. Leave empty if you want to use local mode.
> # export ZEPPELIN_JAVA_OPTS # Additional jvm options. for
> example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g
> -Dspark.cores.max=16"
> # export ZEPPELIN_MEM # Zeppelin jvm mem options Default
> -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
> # export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm
> mem options. Default -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
> # export ZEPPELIN_INTP_JAVA_OPTS # zeppelin interpreter process jvm
> options.
> # export ZEPPELIN_SSL_PORT # ssl port (used when ssl
> environment variable is set to true)
>
> # export ZEPPELIN_LOG_DIR # Where log files are stored. PWD
> by default.
> # export ZEPPELIN_PID_DIR # The pid files are stored.
> ${ZEPPELIN_HOME}/run by default.
> # export ZEPPELIN_WAR_TEMPDIR # The location of jetty temporary
> directory.
> # export ZEPPELIN_NOTEBOOK_DIR # Where notebook saved
> # export ZEPPELIN_NOTEBOOK_HOMESCREEN # Id of notebook to be
> displayed in homescreen. ex) 2A94M5J1Z
> # export ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE # hide homescreen notebook
> from list when this value set to "true". default "false"
> # export ZEPPELIN_NOTEBOOK_S3_BUCKET # Bucket where notebook
> saved
> # export ZEPPELIN_NOTEBOOK_S3_ENDPOINT # Endpoint of the bucket
> # export ZEPPELIN_NOTEBOOK_S3_USER # User in bucket where
> notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json
> # export ZEPPELIN_IDENT_STRING # A string representing this
> instance of zeppelin. $USER by default.
> # export ZEPPELIN_NICENESS # The scheduling priority for
> daemons. Defaults to 0.
> # export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for
> interpreter's additional dependency loading
> # export ZEPPELIN_NOTEBOOK_STORAGE # Refers to pluggable notebook
> storage class, can have two classes simultaneously with a sync between them
> (e.g. local and remote).
> # export ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC # If there are multiple
> notebook storages, should we treat the first one as the only source of
> truth?
>
> #### Spark interpreter configuration ####
>
> ## Use provided spark installation ##
> ## defining SPARK_HOME makes Zeppelin run spark interpreter process
> using spark-submit
> ##
> # export SPARK_HOME # (required) When it
> is defined, load it instead of Zeppelin embedded Spark libraries
> # export SPARK_SUBMIT_OPTIONS # (optional) extra
> options to pass to spark submit. eg) "--driver-memory 512M
> --executor-memory 1G".
> # export SPARK_APP_NAME # (optional) The name
> of spark application.
>
> ## Use embedded spark binaries ##
> ## without SPARK_HOME defined, Zeppelin still able to run spark
> interpreter process using embedded spark binaries.
> ## however, it is not encouraged when you can define SPARK_HOME
> ##
> # Options read in YARN client mode
> export HADOOP_CONF_DIR = /usr/local/lib/hadoop/etc/hadoop/ #
> yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
> # Pyspark (supported with Spark 1.2.1 and above)
> # To configure pyspark, you need to set spark distribution's path to
> 'spark.home' property in Interpreter setting screen in Zeppelin GUI
> # export PYSPARK_PYTHON # path to the python command. must
> be the same path on the driver(Zeppelin) and all workers.
> # export PYTHONPATH
>
> ## Spark interpreter options ##
> ##
> # export ZEPPELIN_SPARK_USEHIVECONTEXT # Use HiveContext instead of
> SQLContext if set true. true by default.
> # export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL
> concurrently if set true. false by default.
> # export ZEPPELIN_SPARK_IMPORTIMPLICIT # Import implicits, UDF
> collection, and sql if set true. true by default.
> # export ZEPPELIN_SPARK_MAXRESULT # Max number of Spark SQL
> result to display. 1000 by default.
> # export ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE # Size in
> characters of the maximum text message to be received by websocket.
> Defaults to 1024000
>
>
> #### HBase interpreter configuration ####
>
> ## To connect to HBase running on a cluster, either HBASE_HOME or
> HBASE_CONF_DIR must be set
>
> # export HBASE_HOME= # (require) Under which HBase
> scripts and configuration should be
> # export HBASE_CONF_DIR= # (optional) Alternatively,
> configuration directory can be set to point to the directory that has
> hbase-site.xml
>
> #### ZeppelinHub connection configuration ####
> # export ZEPPELINHUB_API_ADDRESS # Refers to the address of the
> ZeppelinHub service in use
> # export ZEPPELINHUB_API_TOKEN # Refers to the Zeppelin instance
> token of the user
> # export ZEPPELINHUB_USER_KEY # Optional, when using Zeppelin with
> authentication.
>
>
>
> I also tried simply /usr/local/lib/hadoop and I also create a conf
> directory within /usr/local/lib/hadoop/etc/hadoop and placed
> yarn-site.xml within this folder
>
> Thanks
>
> On Wed, Nov 2, 2016 at 10:06 AM, Hyung Sung Shim <hs...@nflabs.com>
> wrote:
>
>> Could you share your zeppelin-env.sh ?
>> 2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <be...@gmail.com>님이 작성:
>>
>>> Thanks for your reply,
>>> I have tried setting it within zeppelin-env.sh but it doesn't work any
>>> better.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>> Hello.
>>> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/
>>> in the conf/zeppelin-env.sh.
>>> Thanks.
>>> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>>>
>>> Hello,
>>>
>>> I'd like to use zeppelin on my local computer and use it to run spark
>>> executors on a distant yarn cluster since I can't easily install zeppelin
>>> on the cluster gateway.
>>>
>>> I installed the correct hadoop version (2.6), and compiled zeppelin
>>> (from the master branch) as following:
>>>
>>> *mvn clean package -DskipTests -Phadoop-2.6
>>> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>>>
>>> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
>>> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where
>>> the conf files such as yarn-site.xml are). I set
>>> yarn.resourcemanager.hostname to the resource manager of the cluster (I
>>> copied the value from the config file on the cluster) but when I start a
>>> spark command it still tries to connect to 0.0.0.0:8032 as one can see
>>> in the logs:
>>>
>>> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
>>> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
>>> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>> sleepTime=1000 MILLISECONDS)*
>>>
>>> Am I missing something something? Is there any additional parameters to
>>> set?
>>>
>>> Thanks!
>>>
>>> Benoit
>>>
>>>
>>>
>>>
>
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Benoit Hanotte <be...@gmail.com>.
I have only set HADOOP_CONF_DIR as following (my hadoop conf files are in
/usr/local/lib/hadoop/etc/hadoop/, eg
/usr/local/lib/hadoop/etc/hadoop/yarn-site.xml):
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version
2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# export JAVA_HOME=
# export MASTER= # Spark master url. eg.
spark://master_addr:7077. Leave empty if you want to use local mode.
# export ZEPPELIN_JAVA_OPTS # Additional jvm options. for
example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g
-Dspark.cores.max=16"
# export ZEPPELIN_MEM # Zeppelin jvm mem options Default
-Xms1024m -Xmx1024m -XX:MaxPermSize=512m
# export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm mem
options. Default -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
# export ZEPPELIN_INTP_JAVA_OPTS # zeppelin interpreter process jvm
options.
# export ZEPPELIN_SSL_PORT # ssl port (used when ssl environment
variable is set to true)
# export ZEPPELIN_LOG_DIR # Where log files are stored. PWD by
default.
# export ZEPPELIN_PID_DIR # The pid files are stored.
${ZEPPELIN_HOME}/run by default.
# export ZEPPELIN_WAR_TEMPDIR # The location of jetty temporary
directory.
# export ZEPPELIN_NOTEBOOK_DIR # Where notebook saved
# export ZEPPELIN_NOTEBOOK_HOMESCREEN # Id of notebook to be displayed
in homescreen. ex) 2A94M5J1Z
# export ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE # hide homescreen notebook
from list when this value set to "true". default "false"
# export ZEPPELIN_NOTEBOOK_S3_BUCKET # Bucket where notebook
saved
# export ZEPPELIN_NOTEBOOK_S3_ENDPOINT # Endpoint of the bucket
# export ZEPPELIN_NOTEBOOK_S3_USER # User in bucket where
notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json
# export ZEPPELIN_IDENT_STRING # A string representing this instance
of zeppelin. $USER by default.
# export ZEPPELIN_NICENESS # The scheduling priority for daemons.
Defaults to 0.
# export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for
interpreter's additional dependency loading
# export ZEPPELIN_NOTEBOOK_STORAGE # Refers to pluggable notebook
storage class, can have two classes simultaneously with a sync between them
(e.g. local and remote).
# export ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC # If there are multiple
notebook storages, should we treat the first one as the only source of
truth?
#### Spark interpreter configuration ####
## Use provided spark installation ##
## defining SPARK_HOME makes Zeppelin run spark interpreter process
using spark-submit
##
# export SPARK_HOME # (required) When it is
defined, load it instead of Zeppelin embedded Spark libraries
# export SPARK_SUBMIT_OPTIONS # (optional) extra
options to pass to spark submit. eg) "--driver-memory 512M
--executor-memory 1G".
# export SPARK_APP_NAME # (optional) The name
of spark application.
## Use embedded spark binaries ##
## without SPARK_HOME defined, Zeppelin still able to run spark
interpreter process using embedded spark binaries.
## however, it is not encouraged when you can define SPARK_HOME
##
# Options read in YARN client mode
export HADOOP_CONF_DIR = /usr/local/lib/hadoop/etc/hadoop/ #
yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
# Pyspark (supported with Spark 1.2.1 and above)
# To configure pyspark, you need to set spark distribution's path to
'spark.home' property in Interpreter setting screen in Zeppelin GUI
# export PYSPARK_PYTHON # path to the python command. must be
the same path on the driver(Zeppelin) and all workers.
# export PYTHONPATH
## Spark interpreter options ##
##
# export ZEPPELIN_SPARK_USEHIVECONTEXT # Use HiveContext instead of
SQLContext if set true. true by default.
# export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL
concurrently if set true. false by default.
# export ZEPPELIN_SPARK_IMPORTIMPLICIT # Import implicits, UDF
collection, and sql if set true. true by default.
# export ZEPPELIN_SPARK_MAXRESULT # Max number of Spark SQL
result to display. 1000 by default.
# export ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE # Size in
characters of the maximum text message to be received by websocket.
Defaults to 1024000
#### HBase interpreter configuration ####
## To connect to HBase running on a cluster, either HBASE_HOME or
HBASE_CONF_DIR must be set
# export HBASE_HOME= # (require) Under which HBase
scripts and configuration should be
# export HBASE_CONF_DIR= # (optional) Alternatively,
configuration directory can be set to point to the directory that has
hbase-site.xml
#### ZeppelinHub connection configuration ####
# export ZEPPELINHUB_API_ADDRESS # Refers to the address of the
ZeppelinHub service in use
# export ZEPPELINHUB_API_TOKEN # Refers to the Zeppelin instance token
of the user
# export ZEPPELINHUB_USER_KEY # Optional, when using Zeppelin with
authentication.
I also tried simply /usr/local/lib/hadoop and I also create a conf
directory within /usr/local/lib/hadoop/etc/hadoop and placed yarn-site.xml
within this folder
Thanks
On Wed, Nov 2, 2016 at 10:06 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Could you share your zeppelin-env.sh ?
> 2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <be...@gmail.com>님이 작성:
>
>> Thanks for your reply,
>> I have tried setting it within zeppelin-env.sh but it doesn't work any
>> better.
>>
>> Thanks
>>
>> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com>
>> wrote:
>>
>> Hello.
>> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/
>> in the conf/zeppelin-env.sh.
>> Thanks.
>> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>>
>> Hello,
>>
>> I'd like to use zeppelin on my local computer and use it to run spark
>> executors on a distant yarn cluster since I can't easily install zeppelin
>> on the cluster gateway.
>>
>> I installed the correct hadoop version (2.6), and compiled zeppelin (from
>> the master branch) as following:
>>
>> *mvn clean package -DskipTests -Phadoop-2.6
>> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>>
>> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
>> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
>> conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
>> to the resource manager of the cluster (I copied the value from the config
>> file on the cluster) but when I start a spark command it still tries to
>> connect to 0.0.0.0:8032 as one can see in the logs:
>>
>> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
>> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
>> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)*
>>
>> Am I missing something something? Is there any additional parameters to
>> set?
>>
>> Thanks!
>>
>> Benoit
>>
>>
>>
>>
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Could you share your zeppelin-env.sh ?
2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <be...@gmail.com>님이 작성:
> Thanks for your reply,
> I have tried setting it within zeppelin-env.sh but it doesn't work any
> better.
>
> Thanks
>
> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
>
> Hello.
> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/ in
> the conf/zeppelin-env.sh.
> Thanks.
> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>
> Hello,
>
> I'd like to use zeppelin on my local computer and use it to run spark
> executors on a distant yarn cluster since I can't easily install zeppelin
> on the cluster gateway.
>
> I installed the correct hadoop version (2.6), and compiled zeppelin (from
> the master branch) as following:
>
> *mvn clean package -DskipTests -Phadoop-2.6
> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>
> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
> conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
> to the resource manager of the cluster (I copied the value from the config
> file on the cluster) but when I start a spark command it still tries to
> connect to 0.0.0.0:8032 as one can see in the logs:
>
> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
>
> Am I missing something something? Is there any additional parameters to
> set?
>
> Thanks!
>
> Benoit
>
>
>
>
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Benoit Hanotte <be...@gmail.com>.
Thanks for your reply,
I have tried setting it within zeppelin-env.sh but it doesn't work any
better.
Thanks
On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Hello.
> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/
> in the conf/zeppelin-env.sh.
> Thanks.
> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
>
>> Hello,
>>
>> I'd like to use zeppelin on my local computer and use it to run spark
>> executors on a distant yarn cluster since I can't easily install zeppelin
>> on the cluster gateway.
>>
>> I installed the correct hadoop version (2.6), and compiled zeppelin (from
>> the master branch) as following:
>>
>> *mvn clean package -DskipTests -Phadoop-2.6
>> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>>
>> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
>> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
>> conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
>> to the resource manager of the cluster (I copied the value from the config
>> file on the cluster) but when I start a spark command it still tries to
>> connect to 0.0.0.0:8032 as one can see in the logs:
>>
>> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
>> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
>> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)*
>>
>> Am I missing something something? Is there any additional parameters to
>> set?
>>
>> Thanks!
>>
>> Benoit
>>
>>
>>
Re: Zeppelin in local computer using yarn on distant cluster
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Hello.
You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/ in
the conf/zeppelin-env.sh.
Thanks.
2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <be...@gmail.com>님이 작성:
> Hello,
>
> I'd like to use zeppelin on my local computer and use it to run spark
> executors on a distant yarn cluster since I can't easily install zeppelin
> on the cluster gateway.
>
> I installed the correct hadoop version (2.6), and compiled zeppelin (from
> the master branch) as following:
>
> *mvn clean package -DskipTests -Phadoop-2.6
> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>
> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
> conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
> to the resource manager of the cluster (I copied the value from the config
> file on the cluster) but when I start a spark command it still tries to
> connect to 0.0.0.0:8032 as one can see in the logs:
>
> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
>
> Am I missing something something? Is there any additional parameters to
> set?
>
> Thanks!
>
> Benoit
>
>
>