You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Pushkar Raste <pu...@gmail.com> on 2014/10/27 15:54:08 UTC

Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Hi,
I am trying to to deploy memcached package using slider (0.40.0 version). I
keep getting "Wrong FS:
hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:///"
error

I saw that  'Rui Zhang' had similar issue, but she got around by
setting SLIDER_CLASSPATH_EXTRA.
Unfortunately that solution did not work for me. Here is my setup
 1. slider-client.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
href="configuration.xsl"?> <!-- Licensed to the Apache Software Foundation
(ASF) under one or more contributor license agreements. See the NOTICE file
distributed with this work for additional information regarding copyright
ownership. The ASF licenses this file to You under the Apache License:
Version 2.0 (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable
law or agreed to in writing: software distributed under the License is
distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND: either express or implied. See the License for the specific language
governing permissions and limitations under the License. --> <!--
Properties set here are picked up in the client. --> <configuration>
<property> <name>slider.client.resource.origin</name>
<value>conf/slider-client.xml</value> <description>This is just for
diagnostics</description> </property> <property>
<name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>slider.yarn.queue</name> <value>default</value>
<description>YARN queue for the Application Master</description>
</property> <!-- <property> <name>yarn.resourcemanager.address</name>
<value>master:8032</value> </property> <property> <name>fs.defaultFS</name>
<value>hdfs://master:9090</value> </property> <property>
<name>yarn.resourcemanager.principal</name>
<value>yarn/master@MINICLUSTER</value>
</property> <property> <name>slider.security.enabled</name>
<value>true</value> </property> <property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/master@MINICLUSTER</value>
</property> --> <property> <name>yarn.application.classpath</name>
<value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
</property> <property> <name>slider.zookeeper.quorum</name>
<value>localhost:2181</value> </property> <property>
<name>yarn.resourcemanager.address</name> <value>localhost:8032</value>
</property> <property> <name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8030</value> </property> <property>
<name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property>
</configuration>

2. Environment variables in my ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin export
HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL export
HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop



3. Output when I run 'python slider.py version' slider_home =
"/usr/local/slider/slider-0.40" slider_jvm_opts =
"-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
-Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
"/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
'-Djava.awt.headless=true', '-Xmx256m',
'-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
'/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
'org.apache.slider.Slider', 'version'] SLF4J: Class path contains multiple
SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found
binding in
[jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider Core-0.40
Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J: Found
binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled against
Hadoop 2.4.0 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
for an explanation. SLF4J: Actual binding is of type
[org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO
client.SliderClient - Hadoop runtime version branch-2.4.0 with source
checksum 375b2832a6641759c6eaf6e3e998147 and build date 2014-03-31T08:31Z
2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with status 0

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Another observation is this error has no stacktrace. I guess it does happen
during launching the App master. It happens when app master try to launch
container when it tries to localize the resource?

Best,
Siyuan

On Tue, Oct 28, 2014 at 7:02 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> I try to run kafka as slider application
>
>
> On Tue, Oct 28, 2014 at 7:01 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> I had the same problem
>>
>> This is my appConfig.json
>>
>> {
>>   "schema": "http://example.org/specification/v2.0.0",
>>   "metadata": {
>>   },
>>   "global": {
>>     "application.def":
>> "hdfs://localhost:9000/user/siyuan/slider_kafka.zip",
>>     "java_home": "/usr/lib/jvm/java-7-oracle/",
>>     "package_list": "files/kafka_2.10-0.8.1.1.tgz",
>>
>>     "site.global.app_user": "siyuan",
>>     "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/slider_kafka",
>>     "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",
>>
>>     "site.global.additional_cp": "/usr/lib/hadoop/lib/*",
>>     "site.global.xmx_val": "256m",
>>     "site.global.xms_val": "128m"
>>   },
>>   "components": {
>>     "slider-appmaster": {
>>       "jvm.heapsize": "256M"
>>     }
>>   }
>> }
>>
>> On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <jm...@hortonworks.com>
>> wrote:
>>
>>> I think the problem is probably in your appConfig file (e.g. referencing
>>> an application package from the the local filesystem as opposed to HDFS).
>>> Can you attach that?
>>>
>>> — Jon
>>>
>>> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> > I am trying to to deploy memcached package using slider (0.40.0
>>> version). I
>>> > keep getting "Wrong FS:
>>> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
>>> file:///"
>>> > error
>>> >
>>> > I saw that  'Rui Zhang' had similar issue, but she got around by
>>> > setting SLIDER_CLASSPATH_EXTRA.
>>> > Unfortunately that solution did not work for me. Here is my setup
>>> > 1. slider-client.xml
>>> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
>>> > href="configuration.xsl"?> <!-- Licensed to the Apache Software
>>> Foundation
>>> > (ASF) under one or more contributor license agreements. See the NOTICE
>>> file
>>> > distributed with this work for additional information regarding
>>> copyright
>>> > ownership. The ASF licenses this file to You under the Apache License:
>>> > Version 2.0 (the "License"); you may not use this file except in
>>> compliance
>>> > with the License. You may obtain a copy of the License at
>>> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by
>>> applicable
>>> > law or agreed to in writing: software distributed under the License is
>>> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF
>>> ANY
>>> > KIND: either express or implied. See the License for the specific
>>> language
>>> > governing permissions and limitations under the License. --> <!--
>>> > Properties set here are picked up in the client. --> <configuration>
>>> > <property> <name>slider.client.resource.origin</name>
>>> > <value>conf/slider-client.xml</value> <description>This is just for
>>> > diagnostics</description> </property> <property>
>>> > <name>yarn.log-aggregation-enable</name> <value>true</value>
>>> </property>
>>> > <property> <name>slider.yarn.queue</name> <value>default</value>
>>> > <description>YARN queue for the Application Master</description>
>>> > </property> <!-- <property> <name>yarn.resourcemanager.address</name>
>>> > <value>master:8032</value> </property> <property>
>>> <name>fs.defaultFS</name>
>>> > <value>hdfs://master:9090</value> </property> <property>
>>> > <name>yarn.resourcemanager.principal</name>
>>> > <value>yarn/master@MINICLUSTER</value>
>>> > </property> <property> <name>slider.security.enabled</name>
>>> > <value>true</value> </property> <property>
>>> > <name>dfs.namenode.kerberos.principal</name>
>>> > <value>hdfs/master@MINICLUSTER</value>
>>> > </property> --> <property> <name>yarn.application.classpath</name>
>>> >
>>> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
>>> > </property> <property> <name>slider.zookeeper.quorum</name>
>>> > <value>localhost:2181</value> </property> <property>
>>> > <name>yarn.resourcemanager.address</name> <value>localhost:8032</value>
>>> > </property> <property>
>>> <name>yarn.resourcemanager.scheduler.address</name>
>>> > <value>localhost:8030</value> </property> <property>
>>> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value>
>>> </property>
>>> > </configuration>
>>> >
>>> > 2. Environment variables in my ~/.bashrc
>>> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
>>> > HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
>>> > export PATH=$PATH:$HADOOP_INSTALL/sbin export
>>> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
>>> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export
>>> HADOOP_HDFS_HOME=$HADOOP_INSTALL
>>> > export YARN_HOME=$HADOOP_INSTALL export
>>> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
>>> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
>>> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
>>> >
>>> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
>>> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
>>> >
>>> >
>>> >
>>> > 3. Output when I run 'python slider.py version' slider_home =
>>> > "/usr/local/slider/slider-0.40" slider_jvm_opts =
>>> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
>>> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
>>> >
>>> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
>>> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
>>> > '-Djava.awt.headless=true', '-Xmx256m',
>>> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
>>> >
>>> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
>>> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains
>>> multiple
>>> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
>>> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found
>>> > binding in
>>> >
>>> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider
>>> Core-0.40
>>> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J:
>>> Found
>>> > binding in
>>> >
>>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled
>>> against
>>> > Hadoop 2.4.0 SLF4J: See
>>> http://www.slf4j.org/codes.html#multiple_bindings
>>> > for an explanation. SLF4J: Actual binding is of type
>>> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO
>>> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source
>>> > checksum 375b2832a6641759c6eaf6e3e998147 and build date
>>> 2014-03-31T08:31Z
>>> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with
>>> status 0
>>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Sumit Mohanty <sm...@hortonworks.com>.
Sorry, just noticed that you are using 0.40. Let me go through the config
and error.

On Tue, Oct 28, 2014 at 7:40 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:

> Are you using the develop branch or release 0.40?
>
> On Tue, Oct 28, 2014 at 7:02 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> I try to run kafka as slider application
>>
>>
>> On Tue, Oct 28, 2014 at 7:01 PM, hsy541@gmail.com <hs...@gmail.com>
>> wrote:
>>
>> > I had the same problem
>> >
>> > This is my appConfig.json
>> >
>> > {
>> >   "schema": "http://example.org/specification/v2.0.0",
>> >   "metadata": {
>> >   },
>> >   "global": {
>> >     "application.def":
>> > "hdfs://localhost:9000/user/siyuan/slider_kafka.zip",
>> >     "java_home": "/usr/lib/jvm/java-7-oracle/",
>> >     "package_list": "files/kafka_2.10-0.8.1.1.tgz",
>> >
>> >     "site.global.app_user": "siyuan",
>> >     "site.global.app_root":
>> "${AGENT_WORK_ROOT}/app/install/slider_kafka",
>> >     "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",
>> >
>> >     "site.global.additional_cp": "/usr/lib/hadoop/lib/*",
>> >     "site.global.xmx_val": "256m",
>> >     "site.global.xms_val": "128m"
>> >   },
>> >   "components": {
>> >     "slider-appmaster": {
>> >       "jvm.heapsize": "256M"
>> >     }
>> >   }
>> > }
>> >
>> > On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <jm...@hortonworks.com>
>> wrote:
>> >
>> >> I think the problem is probably in your appConfig file (e.g.
>> referencing
>> >> an application package from the the local filesystem as opposed to
>> HDFS).
>> >> Can you attach that?
>> >>
>> >> — Jon
>> >>
>> >> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi,
>> >> > I am trying to to deploy memcached package using slider (0.40.0
>> >> version). I
>> >> > keep getting "Wrong FS:
>> >> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
>> >> file:///"
>> >> > error
>> >> >
>> >> > I saw that  'Rui Zhang' had similar issue, but she got around by
>> >> > setting SLIDER_CLASSPATH_EXTRA.
>> >> > Unfortunately that solution did not work for me. Here is my setup
>> >> > 1. slider-client.xml
>> >> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
>> >> > href="configuration.xsl"?> <!-- Licensed to the Apache Software
>> >> Foundation
>> >> > (ASF) under one or more contributor license agreements. See the
>> NOTICE
>> >> file
>> >> > distributed with this work for additional information regarding
>> >> copyright
>> >> > ownership. The ASF licenses this file to You under the Apache
>> License:
>> >> > Version 2.0 (the "License"); you may not use this file except in
>> >> compliance
>> >> > with the License. You may obtain a copy of the License at
>> >> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by
>> >> applicable
>> >> > law or agreed to in writing: software distributed under the License
>> is
>> >> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF
>> ANY
>> >> > KIND: either express or implied. See the License for the specific
>> >> language
>> >> > governing permissions and limitations under the License. --> <!--
>> >> > Properties set here are picked up in the client. --> <configuration>
>> >> > <property> <name>slider.client.resource.origin</name>
>> >> > <value>conf/slider-client.xml</value> <description>This is just for
>> >> > diagnostics</description> </property> <property>
>> >> > <name>yarn.log-aggregation-enable</name> <value>true</value>
>> </property>
>> >> > <property> <name>slider.yarn.queue</name> <value>default</value>
>> >> > <description>YARN queue for the Application Master</description>
>> >> > </property> <!-- <property> <name>yarn.resourcemanager.address</name>
>> >> > <value>master:8032</value> </property> <property>
>> >> <name>fs.defaultFS</name>
>> >> > <value>hdfs://master:9090</value> </property> <property>
>> >> > <name>yarn.resourcemanager.principal</name>
>> >> > <value>yarn/master@MINICLUSTER</value>
>> >> > </property> <property> <name>slider.security.enabled</name>
>> >> > <value>true</value> </property> <property>
>> >> > <name>dfs.namenode.kerberos.principal</name>
>> >> > <value>hdfs/master@MINICLUSTER</value>
>> >> > </property> --> <property> <name>yarn.application.classpath</name>
>> >> >
>> >>
>> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
>> >> > </property> <property> <name>slider.zookeeper.quorum</name>
>> >> > <value>localhost:2181</value> </property> <property>
>> >> > <name>yarn.resourcemanager.address</name>
>> <value>localhost:8032</value>
>> >> > </property> <property>
>> >> <name>yarn.resourcemanager.scheduler.address</name>
>> >> > <value>localhost:8030</value> </property> <property>
>> >> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value>
>> >> </property>
>> >> > </configuration>
>> >> >
>> >> > 2. Environment variables in my ~/.bashrc
>> >> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
>> >> > HADOOP_INSTALL=/usr/local/hadoop export
>> PATH=$PATH:$HADOOP_INSTALL/bin
>> >> > export PATH=$PATH:$HADOOP_INSTALL/sbin export
>> >> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
>> >> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export
>> >> HADOOP_HDFS_HOME=$HADOOP_INSTALL
>> >> > export YARN_HOME=$HADOOP_INSTALL export
>> >> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
>> >> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
>> >> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
>> >> >
>> >>
>> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
>> >> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
>> >> >
>> >> >
>> >> >
>> >> > 3. Output when I run 'python slider.py version' slider_home =
>> >> > "/usr/local/slider/slider-0.40" slider_jvm_opts =
>> >> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
>> >> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
>> >> >
>> >>
>> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
>> >> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
>> >> > '-Djava.awt.headless=true', '-Xmx256m',
>> >> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
>> >> >
>> >>
>> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
>> >> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains
>> >> multiple
>> >> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
>> >> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J:
>> Found
>> >> > binding in
>> >> >
>> >>
>> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider
>> >> Core-0.40
>> >> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J:
>> Found
>> >> > binding in
>> >> >
>> >>
>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled
>> >> against
>> >> > Hadoop 2.4.0 SLF4J: See
>> >> http://www.slf4j.org/codes.html#multiple_bindings
>> >> > for an explanation. SLF4J: Actual binding is of type
>> >> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main]
>> INFO
>> >> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source
>> >> > checksum 375b2832a6641759c6eaf6e3e998147 and build date
>> >> 2014-03-31T08:31Z
>> >> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with
>> status
>> >> 0
>> >>
>> >>
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to
>> >> which it is addressed and may contain information that is confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >> of this message is not the intended recipient, you are hereby notified
>> >> that
>> >> any printing, copying, dissemination, distribution, disclosure or
>> >> forwarding of this communication is strictly prohibited. If you have
>> >> received this communication in error, please contact the sender
>> >> immediately
>> >> and delete it from your system. Thank You.
>> >>
>> >
>> >
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Sumit Mohanty <sm...@hortonworks.com>.
Are you using the develop branch or release 0.40?

On Tue, Oct 28, 2014 at 7:02 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> I try to run kafka as slider application
>
>
> On Tue, Oct 28, 2014 at 7:01 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > I had the same problem
> >
> > This is my appConfig.json
> >
> > {
> >   "schema": "http://example.org/specification/v2.0.0",
> >   "metadata": {
> >   },
> >   "global": {
> >     "application.def":
> > "hdfs://localhost:9000/user/siyuan/slider_kafka.zip",
> >     "java_home": "/usr/lib/jvm/java-7-oracle/",
> >     "package_list": "files/kafka_2.10-0.8.1.1.tgz",
> >
> >     "site.global.app_user": "siyuan",
> >     "site.global.app_root":
> "${AGENT_WORK_ROOT}/app/install/slider_kafka",
> >     "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",
> >
> >     "site.global.additional_cp": "/usr/lib/hadoop/lib/*",
> >     "site.global.xmx_val": "256m",
> >     "site.global.xms_val": "128m"
> >   },
> >   "components": {
> >     "slider-appmaster": {
> >       "jvm.heapsize": "256M"
> >     }
> >   }
> > }
> >
> > On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <jm...@hortonworks.com>
> wrote:
> >
> >> I think the problem is probably in your appConfig file (e.g. referencing
> >> an application package from the the local filesystem as opposed to
> HDFS).
> >> Can you attach that?
> >>
> >> — Jon
> >>
> >> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I am trying to to deploy memcached package using slider (0.40.0
> >> version). I
> >> > keep getting "Wrong FS:
> >> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
> >> file:///"
> >> > error
> >> >
> >> > I saw that  'Rui Zhang' had similar issue, but she got around by
> >> > setting SLIDER_CLASSPATH_EXTRA.
> >> > Unfortunately that solution did not work for me. Here is my setup
> >> > 1. slider-client.xml
> >> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
> >> > href="configuration.xsl"?> <!-- Licensed to the Apache Software
> >> Foundation
> >> > (ASF) under one or more contributor license agreements. See the NOTICE
> >> file
> >> > distributed with this work for additional information regarding
> >> copyright
> >> > ownership. The ASF licenses this file to You under the Apache License:
> >> > Version 2.0 (the "License"); you may not use this file except in
> >> compliance
> >> > with the License. You may obtain a copy of the License at
> >> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by
> >> applicable
> >> > law or agreed to in writing: software distributed under the License is
> >> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF
> ANY
> >> > KIND: either express or implied. See the License for the specific
> >> language
> >> > governing permissions and limitations under the License. --> <!--
> >> > Properties set here are picked up in the client. --> <configuration>
> >> > <property> <name>slider.client.resource.origin</name>
> >> > <value>conf/slider-client.xml</value> <description>This is just for
> >> > diagnostics</description> </property> <property>
> >> > <name>yarn.log-aggregation-enable</name> <value>true</value>
> </property>
> >> > <property> <name>slider.yarn.queue</name> <value>default</value>
> >> > <description>YARN queue for the Application Master</description>
> >> > </property> <!-- <property> <name>yarn.resourcemanager.address</name>
> >> > <value>master:8032</value> </property> <property>
> >> <name>fs.defaultFS</name>
> >> > <value>hdfs://master:9090</value> </property> <property>
> >> > <name>yarn.resourcemanager.principal</name>
> >> > <value>yarn/master@MINICLUSTER</value>
> >> > </property> <property> <name>slider.security.enabled</name>
> >> > <value>true</value> </property> <property>
> >> > <name>dfs.namenode.kerberos.principal</name>
> >> > <value>hdfs/master@MINICLUSTER</value>
> >> > </property> --> <property> <name>yarn.application.classpath</name>
> >> >
> >>
> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
> >> > </property> <property> <name>slider.zookeeper.quorum</name>
> >> > <value>localhost:2181</value> </property> <property>
> >> > <name>yarn.resourcemanager.address</name>
> <value>localhost:8032</value>
> >> > </property> <property>
> >> <name>yarn.resourcemanager.scheduler.address</name>
> >> > <value>localhost:8030</value> </property> <property>
> >> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value>
> >> </property>
> >> > </configuration>
> >> >
> >> > 2. Environment variables in my ~/.bashrc
> >> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
> >> > HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
> >> > export PATH=$PATH:$HADOOP_INSTALL/sbin export
> >> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
> >> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export
> >> HADOOP_HDFS_HOME=$HADOOP_INSTALL
> >> > export YARN_HOME=$HADOOP_INSTALL export
> >> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
> >> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
> >> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
> >> >
> >>
> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> >> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
> >> >
> >> >
> >> >
> >> > 3. Output when I run 'python slider.py version' slider_home =
> >> > "/usr/local/slider/slider-0.40" slider_jvm_opts =
> >> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
> >> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
> >> >
> >>
> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
> >> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
> >> > '-Djava.awt.headless=true', '-Xmx256m',
> >> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
> >> >
> >>
> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
> >> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains
> >> multiple
> >> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
> >> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J:
> Found
> >> > binding in
> >> >
> >>
> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider
> >> Core-0.40
> >> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J:
> Found
> >> > binding in
> >> >
> >>
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled
> >> against
> >> > Hadoop 2.4.0 SLF4J: See
> >> http://www.slf4j.org/codes.html#multiple_bindings
> >> > for an explanation. SLF4J: Actual binding is of type
> >> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main]
> INFO
> >> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source
> >> > checksum 375b2832a6641759c6eaf6e3e998147 and build date
> >> 2014-03-31T08:31Z
> >> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with
> status
> >> 0
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
I try to run kafka as slider application


On Tue, Oct 28, 2014 at 7:01 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> I had the same problem
>
> This is my appConfig.json
>
> {
>   "schema": "http://example.org/specification/v2.0.0",
>   "metadata": {
>   },
>   "global": {
>     "application.def":
> "hdfs://localhost:9000/user/siyuan/slider_kafka.zip",
>     "java_home": "/usr/lib/jvm/java-7-oracle/",
>     "package_list": "files/kafka_2.10-0.8.1.1.tgz",
>
>     "site.global.app_user": "siyuan",
>     "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/slider_kafka",
>     "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",
>
>     "site.global.additional_cp": "/usr/lib/hadoop/lib/*",
>     "site.global.xmx_val": "256m",
>     "site.global.xms_val": "128m"
>   },
>   "components": {
>     "slider-appmaster": {
>       "jvm.heapsize": "256M"
>     }
>   }
> }
>
> On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <jm...@hortonworks.com> wrote:
>
>> I think the problem is probably in your appConfig file (e.g. referencing
>> an application package from the the local filesystem as opposed to HDFS).
>> Can you attach that?
>>
>> — Jon
>>
>> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com>
>> wrote:
>>
>> > Hi,
>> > I am trying to to deploy memcached package using slider (0.40.0
>> version). I
>> > keep getting "Wrong FS:
>> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
>> file:///"
>> > error
>> >
>> > I saw that  'Rui Zhang' had similar issue, but she got around by
>> > setting SLIDER_CLASSPATH_EXTRA.
>> > Unfortunately that solution did not work for me. Here is my setup
>> > 1. slider-client.xml
>> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
>> > href="configuration.xsl"?> <!-- Licensed to the Apache Software
>> Foundation
>> > (ASF) under one or more contributor license agreements. See the NOTICE
>> file
>> > distributed with this work for additional information regarding
>> copyright
>> > ownership. The ASF licenses this file to You under the Apache License:
>> > Version 2.0 (the "License"); you may not use this file except in
>> compliance
>> > with the License. You may obtain a copy of the License at
>> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by
>> applicable
>> > law or agreed to in writing: software distributed under the License is
>> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> > KIND: either express or implied. See the License for the specific
>> language
>> > governing permissions and limitations under the License. --> <!--
>> > Properties set here are picked up in the client. --> <configuration>
>> > <property> <name>slider.client.resource.origin</name>
>> > <value>conf/slider-client.xml</value> <description>This is just for
>> > diagnostics</description> </property> <property>
>> > <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
>> > <property> <name>slider.yarn.queue</name> <value>default</value>
>> > <description>YARN queue for the Application Master</description>
>> > </property> <!-- <property> <name>yarn.resourcemanager.address</name>
>> > <value>master:8032</value> </property> <property>
>> <name>fs.defaultFS</name>
>> > <value>hdfs://master:9090</value> </property> <property>
>> > <name>yarn.resourcemanager.principal</name>
>> > <value>yarn/master@MINICLUSTER</value>
>> > </property> <property> <name>slider.security.enabled</name>
>> > <value>true</value> </property> <property>
>> > <name>dfs.namenode.kerberos.principal</name>
>> > <value>hdfs/master@MINICLUSTER</value>
>> > </property> --> <property> <name>yarn.application.classpath</name>
>> >
>> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
>> > </property> <property> <name>slider.zookeeper.quorum</name>
>> > <value>localhost:2181</value> </property> <property>
>> > <name>yarn.resourcemanager.address</name> <value>localhost:8032</value>
>> > </property> <property>
>> <name>yarn.resourcemanager.scheduler.address</name>
>> > <value>localhost:8030</value> </property> <property>
>> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value>
>> </property>
>> > </configuration>
>> >
>> > 2. Environment variables in my ~/.bashrc
>> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
>> > HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
>> > export PATH=$PATH:$HADOOP_INSTALL/sbin export
>> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
>> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export
>> HADOOP_HDFS_HOME=$HADOOP_INSTALL
>> > export YARN_HOME=$HADOOP_INSTALL export
>> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
>> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
>> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
>> >
>> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
>> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
>> >
>> >
>> >
>> > 3. Output when I run 'python slider.py version' slider_home =
>> > "/usr/local/slider/slider-0.40" slider_jvm_opts =
>> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
>> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
>> >
>> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
>> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
>> > '-Djava.awt.headless=true', '-Xmx256m',
>> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
>> >
>> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
>> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains
>> multiple
>> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
>> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found
>> > binding in
>> >
>> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider
>> Core-0.40
>> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J: Found
>> > binding in
>> >
>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled
>> against
>> > Hadoop 2.4.0 SLF4J: See
>> http://www.slf4j.org/codes.html#multiple_bindings
>> > for an explanation. SLF4J: Actual binding is of type
>> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO
>> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source
>> > checksum 375b2832a6641759c6eaf6e3e998147 and build date
>> 2014-03-31T08:31Z
>> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with status
>> 0
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
I had the same problem

This is my appConfig.json

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "application.def": "hdfs://localhost:9000/user/siyuan/slider_kafka.zip",
    "java_home": "/usr/lib/jvm/java-7-oracle/",
    "package_list": "files/kafka_2.10-0.8.1.1.tgz",

    "site.global.app_user": "siyuan",
    "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/slider_kafka",
    "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",

    "site.global.additional_cp": "/usr/lib/hadoop/lib/*",
    "site.global.xmx_val": "256m",
    "site.global.xms_val": "128m"
  },
  "components": {
    "slider-appmaster": {
      "jvm.heapsize": "256M"
    }
  }
}

On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <jm...@hortonworks.com> wrote:

> I think the problem is probably in your appConfig file (e.g. referencing
> an application package from the the local filesystem as opposed to HDFS).
> Can you attach that?
>
> — Jon
>
> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com>
> wrote:
>
> > Hi,
> > I am trying to to deploy memcached package using slider (0.40.0
> version). I
> > keep getting "Wrong FS:
> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:///"
> > error
> >
> > I saw that  'Rui Zhang' had similar issue, but she got around by
> > setting SLIDER_CLASSPATH_EXTRA.
> > Unfortunately that solution did not work for me. Here is my setup
> > 1. slider-client.xml
> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
> > href="configuration.xsl"?> <!-- Licensed to the Apache Software
> Foundation
> > (ASF) under one or more contributor license agreements. See the NOTICE
> file
> > distributed with this work for additional information regarding copyright
> > ownership. The ASF licenses this file to You under the Apache License:
> > Version 2.0 (the "License"); you may not use this file except in
> compliance
> > with the License. You may obtain a copy of the License at
> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable
> > law or agreed to in writing: software distributed under the License is
> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF ANY
> > KIND: either express or implied. See the License for the specific
> language
> > governing permissions and limitations under the License. --> <!--
> > Properties set here are picked up in the client. --> <configuration>
> > <property> <name>slider.client.resource.origin</name>
> > <value>conf/slider-client.xml</value> <description>This is just for
> > diagnostics</description> </property> <property>
> > <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> > <property> <name>slider.yarn.queue</name> <value>default</value>
> > <description>YARN queue for the Application Master</description>
> > </property> <!-- <property> <name>yarn.resourcemanager.address</name>
> > <value>master:8032</value> </property> <property>
> <name>fs.defaultFS</name>
> > <value>hdfs://master:9090</value> </property> <property>
> > <name>yarn.resourcemanager.principal</name>
> > <value>yarn/master@MINICLUSTER</value>
> > </property> <property> <name>slider.security.enabled</name>
> > <value>true</value> </property> <property>
> > <name>dfs.namenode.kerberos.principal</name>
> > <value>hdfs/master@MINICLUSTER</value>
> > </property> --> <property> <name>yarn.application.classpath</name>
> >
> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
> > </property> <property> <name>slider.zookeeper.quorum</name>
> > <value>localhost:2181</value> </property> <property>
> > <name>yarn.resourcemanager.address</name> <value>localhost:8032</value>
> > </property> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> > <value>localhost:8030</value> </property> <property>
> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value>
> </property>
> > </configuration>
> >
> > 2. Environment variables in my ~/.bashrc
> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
> > HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
> > export PATH=$PATH:$HADOOP_INSTALL/sbin export
> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export
> HADOOP_HDFS_HOME=$HADOOP_INSTALL
> > export YARN_HOME=$HADOOP_INSTALL export
> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
> >
> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
> >
> >
> >
> > 3. Output when I run 'python slider.py version' slider_home =
> > "/usr/local/slider/slider-0.40" slider_jvm_opts =
> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
> >
> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
> > '-Djava.awt.headless=true', '-Xmx256m',
> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
> >
> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains
> multiple
> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found
> > binding in
> >
> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider
> Core-0.40
> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J: Found
> > binding in
> >
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled
> against
> > Hadoop 2.4.0 SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings
> > for an explanation. SLF4J: Actual binding is of type
> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO
> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source
> > checksum 375b2832a6641759c6eaf6e3e998147 and build date 2014-03-31T08:31Z
> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with status 0
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Jon Maron <jm...@hortonworks.com>.
I think the problem is probably in your appConfig file (e.g. referencing an application package from the the local filesystem as opposed to HDFS).  Can you attach that?

— Jon

On Oct 27, 2014, at 10:54 AM, Pushkar Raste <pu...@gmail.com> wrote:

> Hi,
> I am trying to to deploy memcached package using slider (0.40.0 version). I
> keep getting "Wrong FS:
> hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:///"
> error
> 
> I saw that  'Rui Zhang' had similar issue, but she got around by
> setting SLIDER_CLASSPATH_EXTRA.
> Unfortunately that solution did not work for me. Here is my setup
> 1. slider-client.xml
> <?xml version="1.0"?> <?xml-stylesheet type="text/xsl"
> href="configuration.xsl"?> <!-- Licensed to the Apache Software Foundation
> (ASF) under one or more contributor license agreements. See the NOTICE file
> distributed with this work for additional information regarding copyright
> ownership. The ASF licenses this file to You under the Apache License:
> Version 2.0 (the "License"); you may not use this file except in compliance
> with the License. You may obtain a copy of the License at
> http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable
> law or agreed to in writing: software distributed under the License is
> distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF ANY
> KIND: either express or implied. See the License for the specific language
> governing permissions and limitations under the License. --> <!--
> Properties set here are picked up in the client. --> <configuration>
> <property> <name>slider.client.resource.origin</name>
> <value>conf/slider-client.xml</value> <description>This is just for
> diagnostics</description> </property> <property>
> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>slider.yarn.queue</name> <value>default</value>
> <description>YARN queue for the Application Master</description>
> </property> <!-- <property> <name>yarn.resourcemanager.address</name>
> <value>master:8032</value> </property> <property> <name>fs.defaultFS</name>
> <value>hdfs://master:9090</value> </property> <property>
> <name>yarn.resourcemanager.principal</name>
> <value>yarn/master@MINICLUSTER</value>
> </property> <property> <name>slider.security.enabled</name>
> <value>true</value> </property> <property>
> <name>dfs.namenode.kerberos.principal</name>
> <value>hdfs/master@MINICLUSTER</value>
> </property> --> <property> <name>yarn.application.classpath</name>
> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value>
> </property> <property> <name>slider.zookeeper.quorum</name>
> <value>localhost:2181</value> </property> <property>
> <name>yarn.resourcemanager.address</name> <value>localhost:8032</value>
> </property> <property> <name>yarn.resourcemanager.scheduler.address</name>
> <value>localhost:8030</value> </property> <property>
> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property>
> </configuration>
> 
> 2. Environment variables in my ~/.bashrc
> export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export
> HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin
> export PATH=$PATH:$HADOOP_INSTALL/sbin export
> HADOOP_MAPRED_HOME=$HADOOP_INSTALL export
> HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL
> export YARN_HOME=$HADOOP_INSTALL export
> HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export
> HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export
> HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export
> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
> 
> 
> 
> 3. Output when I run 'python slider.py version' slider_home =
> "/usr/local/slider/slider-0.40" slider_jvm_opts =
> "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m
> -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath =
> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*"
> ready to exec : ['java', '-Djava.net.preferIPv4Stack=true',
> '-Djava.awt.headless=true', '-Xmx256m',
> '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath',
> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*',
> 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains multiple
> SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy -
> Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found
> binding in
> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider Core-0.40
> Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J: Found
> binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled against
> Hadoop 2.4.0 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
> for an explanation. SLF4J: Actual binding is of type
> [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO
> client.SliderClient - Hadoop runtime version branch-2.4.0 with source
> checksum 375b2832a6641759c6eaf6e3e998147 and build date 2014-03-31T08:31Z
> 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with status 0


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Sorry, my classpath should be $HADOOP_HOME/etc/hadoop/, thanks for you
guys' help!



On Wed, Oct 29, 2014 at 12:06 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

>
>
> And all I have in slider-err.txt is
>
> log4j:WARN No appenders could be found for logger (org.apache.slider.core.main.ServiceLauncher).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Exception: Wrong FS: hdfs://localhost:9000/user/siyuan/.slider/cluster/cl15, expected: file:///
>
> Looks like there is no log4j log at all.
>
> How do I properly setup log4j?
>
>
> Best,
>
> Siyuan
>
>
>
> On Wed, Oct 29, 2014 at 11:47 AM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I installed the apache hadoop with just unzip the tar ball.
>> The configuration should be under $HADOOP_HOME/etc/hadoop/, right?
>> I have added $HADOOP_HOME/etc/hadoop/* in yarn.application.classpath
>>
>> On Wed, Oct 29, 2014 at 11:24 AM, Sumit Mohanty <smohanty@hortonworks.com
>> > wrote:
>>
>>> Add the hadoop conf dir as is to yarn classpath variable.
>>>
>>> E.g.
>>> <property>
>>>       <name>yarn.application.classpath</name>
>>>       <value>*/etc/hadoop/conf*,/usr/hdp/current/hadoop-client/*
>>>
>>> On Wed, Oct 29, 2014 at 11:18 AM, hsy541@gmail.com <hs...@gmail.com>
>>> wrote:
>>>
>>> > @Pushkar
>>> >
>>> > Thanks, but it doesn't work for me.
>>> > My slider-client.xml setting is
>>> >
>>> >   <property>
>>> >     <name>yarn.application.classpath</name
>>> >
>>> >
>>> <value>/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/hadoop/common/*,/home/siyuan/hadoop/current/share/hadoop/common/lib/*,/home/siyuan/hadoop/current/share/hadoop/hdfs/lib/*,/home/siyuan/hadoop/current/share/hadoop/yarn/*,/home/siyuan/hadoop/current/share/hadoop/yarn/lib/*</value>
>>> >   </property>
>>> >
>>> > And
>>> >
>>> > echo $HADOOP_CONF_DIR
>>> > /home/siyuan/hadoop/current/etc/hadoop
>>> >
>>> > echo $YARN_CONF_DIR
>>> > /home/siyuan/hadoop/current/etc/hadoop
>>> >
>>> >
>>> >
>>> > Best,
>>> > Siyuan
>>> >
>>> > On Wed, Oct 29, 2014 at 8:48 AM, Pushkar Raste <
>>> pushkar.raste@gmail.com>
>>> > wrote:
>>> >
>>> > > @hsy541 -
>>> > > This could be problem with your hadoop setup. I too had this problem
>>> and
>>> > > thought it was problem with Slider. Turned out that I hadn't
>>> setfollowing
>>> > > variable
>>> > > HADOOP_CONF_DIR
>>> > > YARN_CONF_DIR
>>> > >
>>> > >
>>> > > One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are
>>> > being
>>> > > read in Configuration, It to dump it using Configuration object from
>>> > > SliderAppMaster class. Here is what I added to getClusterFS()
>>> function in
>>> > > SliderAppMaster.java
>>> > >
>>> > > public SliderFileSystem getClusterFS() throws IOException {
>>> > >    log.info("Config is {}", conf.toString());
>>> > >    return new SliderFileSystem(conf);
>>> > >
>>> > > }
>>> > >
>>> > >
>>> > > Oputput clearly indicated me that my core-site.xml yarn-site.xml and
>>> > > hdfs-site.xml were not being read. On further investigation I
>>> realized
>>> > that
>>> > > my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.
>>> > >
>>> > > On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <
>>> stevel@hortonworks.com>
>>> > > wrote:
>>> > >
>>> > > > On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > > > I am trying to to deploy memcached package using slider (0.40.0
>>> > > > version). I
>>> > > > > keep getting "Wrong FS:
>>> > > > > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
>>> > > file:///"
>>> > > > > error
>>> > > > >
>>> > > >
>>> > > >
>>> > > > That's the warning hadoop gives when the filesystem is different
>>> from
>>> > the
>>> > > > path URL handed in. Here something thinks the filesystem is
>>> file://,
>>> > but
>>> > > > the path to the cluster is
>>> > > > hdfs://localhost:9000/user/root/.slider/cluster/c100
>>> > > >
>>> > > > Now, file:// is the default filesystem of Hadoop —its presence is
>>> > warning
>>> > > > sign that that the configuration that the application has loaded
>>> is not
>>> > > > picking up the core-site.xml of that cluster.
>>> > > >
>>> > > > We've been changing bin/slider & slider.py to pick up the local
>>> Hadoop
>>> > > > config dirs if present, but I suspect this message may actually be
>>> > coming
>>> > > > from the deployed slider application master. If that's the case,
>>> the
>>> > > > probable cause is that whatever classpath it is using to pick up
>>> the
>>> > > hadoop
>>> > > > JARs, this path does not include the Hadoop configuration
>>> directory.
>>> > > >
>>> > > > --
>>> > > > CONFIDENTIALITY NOTICE
>>> > > > NOTICE: This message is intended for the use of the individual or
>>> > entity
>>> > > to
>>> > > > which it is addressed and may contain information that is
>>> confidential,
>>> > > > privileged and exempt from disclosure under applicable law. If the
>>> > reader
>>> > > > of this message is not the intended recipient, you are hereby
>>> notified
>>> > > that
>>> > > > any printing, copying, dissemination, distribution, disclosure or
>>> > > > forwarding of this communication is strictly prohibited. If you
>>> have
>>> > > > received this communication in error, please contact the sender
>>> > > immediately
>>> > > > and delete it from your system. Thank You.
>>> > > >
>>> > >
>>> >
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
And all I have in slider-err.txt is

log4j:WARN No appenders could be found for logger
(org.apache.slider.core.main.ServiceLauncher).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
Exception: Wrong FS:
hdfs://localhost:9000/user/siyuan/.slider/cluster/cl15, expected:
file:///

Looks like there is no log4j log at all.

How do I properly setup log4j?


Best,

Siyuan



On Wed, Oct 29, 2014 at 11:47 AM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Hi,
>
> I installed the apache hadoop with just unzip the tar ball.
> The configuration should be under $HADOOP_HOME/etc/hadoop/, right?
> I have added $HADOOP_HOME/etc/hadoop/* in yarn.application.classpath
>
> On Wed, Oct 29, 2014 at 11:24 AM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
>> Add the hadoop conf dir as is to yarn classpath variable.
>>
>> E.g.
>> <property>
>>       <name>yarn.application.classpath</name>
>>       <value>*/etc/hadoop/conf*,/usr/hdp/current/hadoop-client/*
>>
>> On Wed, Oct 29, 2014 at 11:18 AM, hsy541@gmail.com <hs...@gmail.com>
>> wrote:
>>
>> > @Pushkar
>> >
>> > Thanks, but it doesn't work for me.
>> > My slider-client.xml setting is
>> >
>> >   <property>
>> >     <name>yarn.application.classpath</name
>> >
>> >
>> <value>/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/hadoop/common/*,/home/siyuan/hadoop/current/share/hadoop/common/lib/*,/home/siyuan/hadoop/current/share/hadoop/hdfs/lib/*,/home/siyuan/hadoop/current/share/hadoop/yarn/*,/home/siyuan/hadoop/current/share/hadoop/yarn/lib/*</value>
>> >   </property>
>> >
>> > And
>> >
>> > echo $HADOOP_CONF_DIR
>> > /home/siyuan/hadoop/current/etc/hadoop
>> >
>> > echo $YARN_CONF_DIR
>> > /home/siyuan/hadoop/current/etc/hadoop
>> >
>> >
>> >
>> > Best,
>> > Siyuan
>> >
>> > On Wed, Oct 29, 2014 at 8:48 AM, Pushkar Raste <pushkar.raste@gmail.com
>> >
>> > wrote:
>> >
>> > > @hsy541 -
>> > > This could be problem with your hadoop setup. I too had this problem
>> and
>> > > thought it was problem with Slider. Turned out that I hadn't
>> setfollowing
>> > > variable
>> > > HADOOP_CONF_DIR
>> > > YARN_CONF_DIR
>> > >
>> > >
>> > > One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are
>> > being
>> > > read in Configuration, It to dump it using Configuration object from
>> > > SliderAppMaster class. Here is what I added to getClusterFS()
>> function in
>> > > SliderAppMaster.java
>> > >
>> > > public SliderFileSystem getClusterFS() throws IOException {
>> > >    log.info("Config is {}", conf.toString());
>> > >    return new SliderFileSystem(conf);
>> > >
>> > > }
>> > >
>> > >
>> > > Oputput clearly indicated me that my core-site.xml yarn-site.xml and
>> > > hdfs-site.xml were not being read. On further investigation I realized
>> > that
>> > > my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.
>> > >
>> > > On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <
>> stevel@hortonworks.com>
>> > > wrote:
>> > >
>> > > > On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com>
>> > wrote:
>> > > >
>> > > > > I am trying to to deploy memcached package using slider (0.40.0
>> > > > version). I
>> > > > > keep getting "Wrong FS:
>> > > > > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
>> > > file:///"
>> > > > > error
>> > > > >
>> > > >
>> > > >
>> > > > That's the warning hadoop gives when the filesystem is different
>> from
>> > the
>> > > > path URL handed in. Here something thinks the filesystem is file://,
>> > but
>> > > > the path to the cluster is
>> > > > hdfs://localhost:9000/user/root/.slider/cluster/c100
>> > > >
>> > > > Now, file:// is the default filesystem of Hadoop —its presence is
>> > warning
>> > > > sign that that the configuration that the application has loaded is
>> not
>> > > > picking up the core-site.xml of that cluster.
>> > > >
>> > > > We've been changing bin/slider & slider.py to pick up the local
>> Hadoop
>> > > > config dirs if present, but I suspect this message may actually be
>> > coming
>> > > > from the deployed slider application master. If that's the case, the
>> > > > probable cause is that whatever classpath it is using to pick up the
>> > > hadoop
>> > > > JARs, this path does not include the Hadoop configuration directory.
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > to
>> > > > which it is addressed and may contain information that is
>> confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Hi,

I installed the apache hadoop with just unzip the tar ball.
The configuration should be under $HADOOP_HOME/etc/hadoop/, right?
I have added $HADOOP_HOME/etc/hadoop/* in yarn.application.classpath

On Wed, Oct 29, 2014 at 11:24 AM, Sumit Mohanty <sm...@hortonworks.com>
wrote:

> Add the hadoop conf dir as is to yarn classpath variable.
>
> E.g.
> <property>
>       <name>yarn.application.classpath</name>
>       <value>*/etc/hadoop/conf*,/usr/hdp/current/hadoop-client/*
>
> On Wed, Oct 29, 2014 at 11:18 AM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > @Pushkar
> >
> > Thanks, but it doesn't work for me.
> > My slider-client.xml setting is
> >
> >   <property>
> >     <name>yarn.application.classpath</name
> >
> >
> <value>/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/hadoop/common/*,/home/siyuan/hadoop/current/share/hadoop/common/lib/*,/home/siyuan/hadoop/current/share/hadoop/hdfs/lib/*,/home/siyuan/hadoop/current/share/hadoop/yarn/*,/home/siyuan/hadoop/current/share/hadoop/yarn/lib/*</value>
> >   </property>
> >
> > And
> >
> > echo $HADOOP_CONF_DIR
> > /home/siyuan/hadoop/current/etc/hadoop
> >
> > echo $YARN_CONF_DIR
> > /home/siyuan/hadoop/current/etc/hadoop
> >
> >
> >
> > Best,
> > Siyuan
> >
> > On Wed, Oct 29, 2014 at 8:48 AM, Pushkar Raste <pu...@gmail.com>
> > wrote:
> >
> > > @hsy541 -
> > > This could be problem with your hadoop setup. I too had this problem
> and
> > > thought it was problem with Slider. Turned out that I hadn't
> setfollowing
> > > variable
> > > HADOOP_CONF_DIR
> > > YARN_CONF_DIR
> > >
> > >
> > > One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are
> > being
> > > read in Configuration, It to dump it using Configuration object from
> > > SliderAppMaster class. Here is what I added to getClusterFS() function
> in
> > > SliderAppMaster.java
> > >
> > > public SliderFileSystem getClusterFS() throws IOException {
> > >    log.info("Config is {}", conf.toString());
> > >    return new SliderFileSystem(conf);
> > >
> > > }
> > >
> > >
> > > Oputput clearly indicated me that my core-site.xml yarn-site.xml and
> > > hdfs-site.xml were not being read. On further investigation I realized
> > that
> > > my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.
> > >
> > > On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <
> stevel@hortonworks.com>
> > > wrote:
> > >
> > > > On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com>
> > wrote:
> > > >
> > > > > I am trying to to deploy memcached package using slider (0.40.0
> > > > version). I
> > > > > keep getting "Wrong FS:
> > > > > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
> > > file:///"
> > > > > error
> > > > >
> > > >
> > > >
> > > > That's the warning hadoop gives when the filesystem is different from
> > the
> > > > path URL handed in. Here something thinks the filesystem is file://,
> > but
> > > > the path to the cluster is
> > > > hdfs://localhost:9000/user/root/.slider/cluster/c100
> > > >
> > > > Now, file:// is the default filesystem of Hadoop —its presence is
> > warning
> > > > sign that that the configuration that the application has loaded is
> not
> > > > picking up the core-site.xml of that cluster.
> > > >
> > > > We've been changing bin/slider & slider.py to pick up the local
> Hadoop
> > > > config dirs if present, but I suspect this message may actually be
> > coming
> > > > from the deployed slider application master. If that's the case, the
> > > > probable cause is that whatever classpath it is using to pick up the
> > > hadoop
> > > > JARs, this path does not include the Hadoop configuration directory.
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Sumit Mohanty <sm...@hortonworks.com>.
Add the hadoop conf dir as is to yarn classpath variable.

E.g.
<property>
      <name>yarn.application.classpath</name>
      <value>*/etc/hadoop/conf*,/usr/hdp/current/hadoop-client/*

On Wed, Oct 29, 2014 at 11:18 AM, hsy541@gmail.com <hs...@gmail.com> wrote:

> @Pushkar
>
> Thanks, but it doesn't work for me.
> My slider-client.xml setting is
>
>   <property>
>     <name>yarn.application.classpath</name
>
> <value>/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/hadoop/common/*,/home/siyuan/hadoop/current/share/hadoop/common/lib/*,/home/siyuan/hadoop/current/share/hadoop/hdfs/lib/*,/home/siyuan/hadoop/current/share/hadoop/yarn/*,/home/siyuan/hadoop/current/share/hadoop/yarn/lib/*</value>
>   </property>
>
> And
>
> echo $HADOOP_CONF_DIR
> /home/siyuan/hadoop/current/etc/hadoop
>
> echo $YARN_CONF_DIR
> /home/siyuan/hadoop/current/etc/hadoop
>
>
>
> Best,
> Siyuan
>
> On Wed, Oct 29, 2014 at 8:48 AM, Pushkar Raste <pu...@gmail.com>
> wrote:
>
> > @hsy541 -
> > This could be problem with your hadoop setup. I too had this problem and
> > thought it was problem with Slider. Turned out that I hadn't setfollowing
> > variable
> > HADOOP_CONF_DIR
> > YARN_CONF_DIR
> >
> >
> > One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are
> being
> > read in Configuration, It to dump it using Configuration object from
> > SliderAppMaster class. Here is what I added to getClusterFS() function in
> > SliderAppMaster.java
> >
> > public SliderFileSystem getClusterFS() throws IOException {
> >    log.info("Config is {}", conf.toString());
> >    return new SliderFileSystem(conf);
> >
> > }
> >
> >
> > Oputput clearly indicated me that my core-site.xml yarn-site.xml and
> > hdfs-site.xml were not being read. On further investigation I realized
> that
> > my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.
> >
> > On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com>
> wrote:
> > >
> > > > I am trying to to deploy memcached package using slider (0.40.0
> > > version). I
> > > > keep getting "Wrong FS:
> > > > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
> > file:///"
> > > > error
> > > >
> > >
> > >
> > > That's the warning hadoop gives when the filesystem is different from
> the
> > > path URL handed in. Here something thinks the filesystem is file://,
> but
> > > the path to the cluster is
> > > hdfs://localhost:9000/user/root/.slider/cluster/c100
> > >
> > > Now, file:// is the default filesystem of Hadoop —its presence is
> warning
> > > sign that that the configuration that the application has loaded is not
> > > picking up the core-site.xml of that cluster.
> > >
> > > We've been changing bin/slider & slider.py to pick up the local Hadoop
> > > config dirs if present, but I suspect this message may actually be
> coming
> > > from the deployed slider application master. If that's the case, the
> > > probable cause is that whatever classpath it is using to pick up the
> > hadoop
> > > JARs, this path does not include the Hadoop configuration directory.
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
@Pushkar

Thanks, but it doesn't work for me.
My slider-client.xml setting is

  <property>
    <name>yarn.application.classpath</name
<value>/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/hadoop/common/*,/home/siyuan/hadoop/current/share/hadoop/common/lib/*,/home/siyuan/hadoop/current/share/hadoop/hdfs/lib/*,/home/siyuan/hadoop/current/share/hadoop/yarn/*,/home/siyuan/hadoop/current/share/hadoop/yarn/lib/*</value>
  </property>

And

echo $HADOOP_CONF_DIR
/home/siyuan/hadoop/current/etc/hadoop

echo $YARN_CONF_DIR
/home/siyuan/hadoop/current/etc/hadoop



Best,
Siyuan

On Wed, Oct 29, 2014 at 8:48 AM, Pushkar Raste <pu...@gmail.com>
wrote:

> @hsy541 -
> This could be problem with your hadoop setup. I too had this problem and
> thought it was problem with Slider. Turned out that I hadn't setfollowing
> variable
> HADOOP_CONF_DIR
> YARN_CONF_DIR
>
>
> One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are being
> read in Configuration, It to dump it using Configuration object from
> SliderAppMaster class. Here is what I added to getClusterFS() function in
> SliderAppMaster.java
>
> public SliderFileSystem getClusterFS() throws IOException {
>    log.info("Config is {}", conf.toString());
>    return new SliderFileSystem(conf);
>
> }
>
>
> Oputput clearly indicated me that my core-site.xml yarn-site.xml and
> hdfs-site.xml were not being read. On further investigation I realized that
> my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.
>
> On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com> wrote:
> >
> > > I am trying to to deploy memcached package using slider (0.40.0
> > version). I
> > > keep getting "Wrong FS:
> > > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected:
> file:///"
> > > error
> > >
> >
> >
> > That's the warning hadoop gives when the filesystem is different from the
> > path URL handed in. Here something thinks the filesystem is file://, but
> > the path to the cluster is
> > hdfs://localhost:9000/user/root/.slider/cluster/c100
> >
> > Now, file:// is the default filesystem of Hadoop —its presence is warning
> > sign that that the configuration that the application has loaded is not
> > picking up the core-site.xml of that cluster.
> >
> > We've been changing bin/slider & slider.py to pick up the local Hadoop
> > config dirs if present, but I suspect this message may actually be coming
> > from the deployed slider application master. If that's the case, the
> > probable cause is that whatever classpath it is using to pick up the
> hadoop
> > JARs, this path does not include the Hadoop configuration directory.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Pushkar Raste <pu...@gmail.com>.
@hsy541 -
This could be problem with your hadoop setup. I too had this problem and
thought it was problem with Slider. Turned out that I hadn't setfollowing
variable
HADOOP_CONF_DIR
YARN_CONF_DIR


One way to verify if your 'core-site.xml' and 'yar-site.xml' ... are being
read in Configuration, It to dump it using Configuration object from
SliderAppMaster class. Here is what I added to getClusterFS() function in
SliderAppMaster.java

public SliderFileSystem getClusterFS() throws IOException {
   log.info("Config is {}", conf.toString());
   return new SliderFileSystem(conf);

}


Oputput clearly indicated me that my core-site.xml yarn-site.xml and
hdfs-site.xml were not being read. On further investigation I realized that
my HADOOP_CONF_DIR and YARN_CONF_DIR were not set.

On Wed, Oct 29, 2014 at 8:32 AM, Steve Loughran <st...@hortonworks.com>
wrote:

> On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com> wrote:
>
> > I am trying to to deploy memcached package using slider (0.40.0
> version). I
> > keep getting "Wrong FS:
> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:///"
> > error
> >
>
>
> That's the warning hadoop gives when the filesystem is different from the
> path URL handed in. Here something thinks the filesystem is file://, but
> the path to the cluster is
> hdfs://localhost:9000/user/root/.slider/cluster/c100
>
> Now, file:// is the default filesystem of Hadoop —its presence is warning
> sign that that the configuration that the application has loaded is not
> picking up the core-site.xml of that cluster.
>
> We've been changing bin/slider & slider.py to pick up the local Hadoop
> config dirs if present, but I suspect this message may actually be coming
> from the deployed slider application master. If that's the case, the
> probable cause is that whatever classpath it is using to pick up the hadoop
> JARs, this path does not include the Hadoop configuration directory.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Posted by Steve Loughran <st...@hortonworks.com>.
On 27 October 2014 14:54, Pushkar Raste <pu...@gmail.com> wrote:

> I am trying to to deploy memcached package using slider (0.40.0 version). I
> keep getting "Wrong FS:
> hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:///"
> error
>


That's the warning hadoop gives when the filesystem is different from the
path URL handed in. Here something thinks the filesystem is file://, but
the path to the cluster is
hdfs://localhost:9000/user/root/.slider/cluster/c100

Now, file:// is the default filesystem of Hadoop —its presence is warning
sign that that the configuration that the application has loaded is not
picking up the core-site.xml of that cluster.

We've been changing bin/slider & slider.py to pick up the local Hadoop
config dirs if present, but I suspect this message may actually be coming
from the deployed slider application master. If that's the case, the
probable cause is that whatever classpath it is using to pick up the hadoop
JARs, this path does not include the Hadoop configuration directory.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.