You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Alexandru Calin <al...@gmail.com> on 2015/03/08 20:15:27 UTC
Another 1/1 local-dirs are bad
I have two hadoop instances running inside two lxc containers on the same
host, a hadoop-master and a hadoop-slave1.
While starting YARN & DFS on master I get this UNHEALTHY state for
hadoop-slave1.
For what I've found on the web it must be one of these two possibilities:
a. Not enough disk space.
b. Permission issue.
a. df -h says otherwise :
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 91G 68G 19G 79% /
none 4,0K 0 4,0K 0% /sys/fs/cgroup
udev 3,8G 4,0K 3,8G 1% /dev
tmpfs 769M 1,3M 768M 1% /run
none 5,0M 0 5,0M 0% /run/lock
none 3,8G 536K 3,8G 1% /run/shm
none 100M 48K 100M 1% /run/user
b.
ll /usr/local/hadoop :
............
drwxr-xr-x 2 hduser hadoop 4096 Mar 8 19:10 local/
ll /usr/local/hadoop/logs :
total 132
drwxr-xr-x 3 hduser hadoop 4096 Mar 8 18:55 ./
drwxr-xr-x 12 hduser hadoop 4096 Mar 8 18:54 ../
-rw-r--r-- 1 hduser hadoop 46222 Mar 8 18:55
hadoop-hduser-datanode-hadoop-slave1.log
-rw-r--r-- 1 hduser hadoop 718 Mar 8 18:55
hadoop-hduser-datanode-hadoop-slave1.out
-rw-r--r-- 1 hduser hadoop 0 Mar 8 17:08 SecurityAuth-hduser.audit
drwxr-xr-x 2 hduser hadoop 4096 Mar 8 19:10 userlogs/
-rw-r--r-- 1 hduser hadoop 56645 Mar 8 19:08
yarn-hduser-nodemanager-hadoop-slave1.log
-rw-r--r-- 1 hduser hadoop 702 Mar 8 18:56
yarn-hduser-nodemanager-hadoop-slave1.out
my yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value> org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop-master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop-master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop-master:8050</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///usr/local/hadoop/local</value>
</property>
</configuration>
And the error is this:
INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node
hadoop-slave1:48673 reported UNHEALTHY with details: 1/1 local-dirs are
bad: /usr/local/hadoop/local; 1/1 log-dirs are bad:
/usr/local/hadoop/logs/userlogs
Any ideas ?
Thanks
SolR integration in HDP
Posted by Ulul <ha...@ulul.org>.
Hi
I will likely get at least one "you should post in solr mailing list" or
"this belongs to Hortonworks forums" answer but I really would like
hadoopers point of view on SolR integration in HDP
SolR is presented by Hortonwoks as an integrated search feature in HDP
but you can hardly say that SolR is a first class citizen in HDP 2.2 :
- No ambari integration (and nothing appears on Ambari roadmap 2.0 or 2.1)
- not even an HDP rpm in repos
- YARN integration is yarn-proto Lucidworks github project that
requested integration through SOLR-6743
<https://issues.apache.org/jira/browse/SOLR-6743> Jira without much
response so far
- only provided doc is a sandbox tutorial
- SolR MapReduce integration is provided through a proprietary
lucidworks jar
Cloudera entreprise, through SolR service, *hue search app, Lily for
HBase integration, seems to provide a better integration.
Am I missing something ? Is SolR integration considered as an entreprise
feature that will require proprietary software on the foreseeable future ?
My current feeling is that, since SolR integration seems rather virtual
or reserved to paying-for-support users, I could as well stick to
Elasticsearch - that I know - since they release MR integration and
Yarn support as OSS and that I can integrate with HBase through Phoenix
and JDBC.
Any thoughts and feedback welcome
Thank you
Ulul
SolR integration in HDP
Posted by Ulul <ha...@ulul.org>.
Hi
I will likely get at least one "you should post in solr mailing list" or
"this belongs to Hortonworks forums" answer but I really would like
hadoopers point of view on SolR integration in HDP
SolR is presented by Hortonwoks as an integrated search feature in HDP
but you can hardly say that SolR is a first class citizen in HDP 2.2 :
- No ambari integration (and nothing appears on Ambari roadmap 2.0 or 2.1)
- not even an HDP rpm in repos
- YARN integration is yarn-proto Lucidworks github project that
requested integration through SOLR-6743
<https://issues.apache.org/jira/browse/SOLR-6743> Jira without much
response so far
- only provided doc is a sandbox tutorial
- SolR MapReduce integration is provided through a proprietary
lucidworks jar
Cloudera entreprise, through SolR service, *hue search app, Lily for
HBase integration, seems to provide a better integration.
Am I missing something ? Is SolR integration considered as an entreprise
feature that will require proprietary software on the foreseeable future ?
My current feeling is that, since SolR integration seems rather virtual
or reserved to paying-for-support users, I could as well stick to
Elasticsearch - that I know - since they release MR integration and
Yarn support as OSS and that I can integrate with HBase through Phoenix
and JDBC.
Any thoughts and feedback welcome
Thank you
Ulul
SolR integration in HDP
Posted by Ulul <ha...@ulul.org>.
Hi
I will likely get at least one "you should post in solr mailing list" or
"this belongs to Hortonworks forums" answer but I really would like
hadoopers point of view on SolR integration in HDP
SolR is presented by Hortonwoks as an integrated search feature in HDP
but you can hardly say that SolR is a first class citizen in HDP 2.2 :
- No ambari integration (and nothing appears on Ambari roadmap 2.0 or 2.1)
- not even an HDP rpm in repos
- YARN integration is yarn-proto Lucidworks github project that
requested integration through SOLR-6743
<https://issues.apache.org/jira/browse/SOLR-6743> Jira without much
response so far
- only provided doc is a sandbox tutorial
- SolR MapReduce integration is provided through a proprietary
lucidworks jar
Cloudera entreprise, through SolR service, *hue search app, Lily for
HBase integration, seems to provide a better integration.
Am I missing something ? Is SolR integration considered as an entreprise
feature that will require proprietary software on the foreseeable future ?
My current feeling is that, since SolR integration seems rather virtual
or reserved to paying-for-support users, I could as well stick to
Elasticsearch - that I know - since they release MR integration and
Yarn support as OSS and that I can integrate with HBase through Phoenix
and JDBC.
Any thoughts and feedback welcome
Thank you
Ulul
SolR integration in HDP
Posted by Ulul <ha...@ulul.org>.
Hi
I will likely get at least one "you should post in solr mailing list" or
"this belongs to Hortonworks forums" answer but I really would like
hadoopers point of view on SolR integration in HDP
SolR is presented by Hortonwoks as an integrated search feature in HDP
but you can hardly say that SolR is a first class citizen in HDP 2.2 :
- No ambari integration (and nothing appears on Ambari roadmap 2.0 or 2.1)
- not even an HDP rpm in repos
- YARN integration is yarn-proto Lucidworks github project that
requested integration through SOLR-6743
<https://issues.apache.org/jira/browse/SOLR-6743> Jira without much
response so far
- only provided doc is a sandbox tutorial
- SolR MapReduce integration is provided through a proprietary
lucidworks jar
Cloudera entreprise, through SolR service, *hue search app, Lily for
HBase integration, seems to provide a better integration.
Am I missing something ? Is SolR integration considered as an entreprise
feature that will require proprietary software on the foreseeable future ?
My current feeling is that, since SolR integration seems rather virtual
or reserved to paying-for-support users, I could as well stick to
Elasticsearch - that I know - since they release MR integration and
Yarn support as OSS and that I can integrate with HBase through Phoenix
and JDBC.
Any thoughts and feedback welcome
Thank you
Ulul