You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Subho Banerjee <su...@gmail.com> on 2012/08/13 06:21:05 UTC
Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Hello,
I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
When running hadoop on pseudo-distributed mode, the map seems to work, but
it cannot compute the reduce.
12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
attempt_201208130857_0001_m_000000_0, Status : FAILED
Too many fetch-failures
12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
returned HTTP response code: 403 for URL:
http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
returned HTTP response code: 403 for URL:
http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
attempt_201208130857_0001_m_000001_0, Status : FAILED
Too many fetch-failures
Here is what I get when I try to see the tasklog using the links given in
the output
http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
--->
2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
SCDynamicStore
http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
--->
I have changed my hadoop-env.sh acoording to Mathew Buckett in
https://issues.apache.org/jira/browse/HADOOP-7489
Also this error of Unable to load realm info from SCDynamicStore does not
show up when I do 'hadoop namenode -format' or 'start-all.sh'
I am also attaching a zipped copy of my logs
Cheers,
Subho.
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Hemanth Yamijala <yh...@gmail.com>.
The mapred.local.dir is local directories on the file system of slave
nodes. In pseudo distributed mode, this would be your own machine. If
you've specified any configuration for it, it should be in your
mapred-site.xml. If not, it defaults to value of
hadoop.tmp.dir/mapred/local. The default value of hadoop.tmp.dir is
/tmp/hadoop-<username>.
Thanks
Hemanth
On Mon, Aug 13, 2012 at 11:42 PM, Subho Banerjee <su...@gmail.com> wrote:
> Where do I set this?
>
>
> On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello Subho,
>>
>> Please check the permission of mapred.local.dir is. This is the
>> place where map outputs are store. Reduce phase is sending the read requests
>> but this directory is not accessible. As a result 403 is thrown.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Hemanth Yamijala <yh...@gmail.com>.
The mapred.local.dir is local directories on the file system of slave
nodes. In pseudo distributed mode, this would be your own machine. If
you've specified any configuration for it, it should be in your
mapred-site.xml. If not, it defaults to value of
hadoop.tmp.dir/mapred/local. The default value of hadoop.tmp.dir is
/tmp/hadoop-<username>.
Thanks
Hemanth
On Mon, Aug 13, 2012 at 11:42 PM, Subho Banerjee <su...@gmail.com> wrote:
> Where do I set this?
>
>
> On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello Subho,
>>
>> Please check the permission of mapred.local.dir is. This is the
>> place where map outputs are store. Reduce phase is sending the read requests
>> but this directory is not accessible. As a result 403 is thrown.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Hemanth Yamijala <yh...@gmail.com>.
The mapred.local.dir is local directories on the file system of slave
nodes. In pseudo distributed mode, this would be your own machine. If
you've specified any configuration for it, it should be in your
mapred-site.xml. If not, it defaults to value of
hadoop.tmp.dir/mapred/local. The default value of hadoop.tmp.dir is
/tmp/hadoop-<username>.
Thanks
Hemanth
On Mon, Aug 13, 2012 at 11:42 PM, Subho Banerjee <su...@gmail.com> wrote:
> Where do I set this?
>
>
> On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello Subho,
>>
>> Please check the permission of mapred.local.dir is. This is the
>> place where map outputs are store. Reduce phase is sending the read requests
>> but this directory is not accessible. As a result 403 is thrown.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Hemanth Yamijala <yh...@gmail.com>.
The mapred.local.dir is local directories on the file system of slave
nodes. In pseudo distributed mode, this would be your own machine. If
you've specified any configuration for it, it should be in your
mapred-site.xml. If not, it defaults to value of
hadoop.tmp.dir/mapred/local. The default value of hadoop.tmp.dir is
/tmp/hadoop-<username>.
Thanks
Hemanth
On Mon, Aug 13, 2012 at 11:42 PM, Subho Banerjee <su...@gmail.com> wrote:
> Where do I set this?
>
>
> On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello Subho,
>>
>> Please check the permission of mapred.local.dir is. This is the
>> place where map outputs are store. Reduce phase is sending the read requests
>> but this directory is not accessible. As a result 403 is thrown.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Subho Banerjee <su...@gmail.com>.
Where do I set this?
On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Subho,
>
> Please check the permission of mapred.local.dir is. This is the
> place where map outputs are store. Reduce phase is sending the read
> requests but this directory is not accessible. As a result 403 is thrown.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Subho Banerjee <su...@gmail.com>.
Where do I set this?
On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Subho,
>
> Please check the permission of mapred.local.dir is. This is the
> place where map outputs are store. Reduce phase is sending the read
> requests but this directory is not accessible. As a result 403 is thrown.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Subho Banerjee <su...@gmail.com>.
Where do I set this?
On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Subho,
>
> Please check the permission of mapred.local.dir is. This is the
> place where map outputs are store. Reduce phase is sending the read
> requests but this directory is not accessible. As a result 403 is thrown.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Subho Banerjee <su...@gmail.com>.
Where do I set this?
On Mon, Aug 13, 2012 at 7:52 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Subho,
>
> Please check the permission of mapred.local.dir is. This is the
> place where map outputs are store. Reduce phase is sending the read
> requests but this directory is not accessible. As a result 403 is thrown.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Subho,
Please check the permission of mapred.local.dir is. This is the
place where map outputs are store. Reduce phase is sending the read
requests but this directory is not accessible. As a result 403 is thrown.
Regards,
Mohammad Tariq
On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Pseudo-distributed mode is good for developing and testing hadoop
code. But instead of experimenting with hadoop on your mac, I would go
for hadoop on EC2. With starcluster http://web.mit.edu/star/cluster/
it takes just a single command to start hadoop. You also get a fixed
environment.
-Håvard
On Mon, Aug 13, 2012 at 6:21 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
No. the content in hadoop-env.sh is like the following:
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK-Djava.security.krb5.kdc=kdc0.ox.ac.uk:
kdc1.ox.ac.uk"
Best Regards,
Stone
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
No. the content in hadoop-env.sh is like the following:
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK-Djava.security.krb5.kdc=kdc0.ox.ac.uk:
kdc1.ox.ac.uk"
Best Regards,
Stone
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
On Fri, Aug 31, 2012 at 3:28 PM, Stone <st...@gmail.com> wrote:
> Fixed this weird issue by change hostname to localhost.
>
> $ hostname
> Stones-MacBook-Air.local
>
> when hostname is Stones-MacBook-Air.local
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
10.18.101.141
> $ scutil --set HostName localhost
> $ hostname
> localhost
>
> when hostname is localhost
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
127.0.0.1
> After changing the hostname restart hadoop. then everything works.
>
> Any ideas about the root cause ?
>
>
> On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Stone,
>>
>> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
>> IPv6?
>>
>> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
>> > FYI : the jstack output for the hanged job:
>> >
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
>> > mode):
>> >
>> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
>> runnable
>> > [109b8f000]
>> > java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> > at java.net.ServerSocket.accept(ServerSocket.java:430)
>> > at
>> >
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>> > at
>> >
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>> > at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000
>> waiting on
>> > condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
>> > condition [109fff000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
>> daemon
>> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
>> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>> >
>> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
>> > condition [109cf6000]
>> > java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for <7c1950890> (a
>> > java.util.concurrent.SynchronousQueue$TransferStack)
>> > at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
>> > runnable [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
>> runnable
>> > [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
>> > nid=0x108c78000 waiting on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
>> Object.wait()
>> > [108a6e000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> >
>> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
>> > Object.wait() [10896b000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
>> > at java.lang.Object.wait(Object.java:485)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
>> > [1008e9000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
>> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>> >
>> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
>> > nid=0x103ced000 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
>> > nid=0x103df0000 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
>> > nid=0x103ef3000 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
>> > nid=0x103ff6000 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000
>> nid=0x1084e7000
>> > runnable
>> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
>> waiting
>> > on condition
>> >
>> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
>> runnable
>> > JNI global references: 1457
>> >
>> >
>> >
>> > Best Regards,
>> > Stone
>> >
>> >
>> >
>> >
>> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>> >>
>> >> I got the same issue today. map tasks finished quickly but reduce is
>> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>> >>
>> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
>> 600
>> >> seconds. Killing!
>> >>
>> >>
>> >>
>> >> logs for the reducer :
>> >>
>> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>> >>
>> >>
>> >>
>> >> stdout logs
>> >>
>> >>
>> >> stderr logs
>> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> >> SCDynamicStore
>> >>
>> >>
>> >> syslog logs
>> >> 2012-08-31 01:13:06,421 INFO
>> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> already
>> >> set up for Hadoop, not re-installing.
>> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> >> Unable to load native-hadoop library for your platform... using
>> builtin-java
>> >> classes where applicable
>> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> >> ResourceCalculatorPlugin : null
>> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> ShuffleRamManager: MemoryLimit=144965632,
>> MaxSingleShuffleLimit=36241408
>> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> merging in
>> >> memory files
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> polling Map
>> >> Completion Events
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 0 is
>> >> already in progress
>> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >>
>> >> I can successfully run the wordcount example on my old macbook (os x
>> 10.6)
>> >> previously.
>> >>
>> >> Any suggestions ?
>> >>
>> >> Best Regards,
>> >> Stone
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>> >>>
>> >>>
>> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> >>> but it cannot compute the reduce.
>> >>>
>> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> >>> job_201208130857_0001
>> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>>
>> >>> Here is what I get when I try to see the tasklog using the links
>> given in
>> >>> the output
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> --->
>> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info
>> from
>> >>> SCDynamicStore
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> --->
>> >>>
>> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> >>> https://issues.apache.org/jira/browse/HADOOP-7489
>> >>>
>> >>> Also this error of Unable to load realm info from SCDynamicStore does
>> not
>> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>> >>>
>> >>> I am also attaching a zipped copy of my logs
>> >>>
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Subho.
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
On Fri, Aug 31, 2012 at 3:28 PM, Stone <st...@gmail.com> wrote:
> Fixed this weird issue by change hostname to localhost.
>
> $ hostname
> Stones-MacBook-Air.local
>
> when hostname is Stones-MacBook-Air.local
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
10.18.101.141
> $ scutil --set HostName localhost
> $ hostname
> localhost
>
> when hostname is localhost
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
127.0.0.1
> After changing the hostname restart hadoop. then everything works.
>
> Any ideas about the root cause ?
>
>
> On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Stone,
>>
>> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
>> IPv6?
>>
>> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
>> > FYI : the jstack output for the hanged job:
>> >
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
>> > mode):
>> >
>> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
>> runnable
>> > [109b8f000]
>> > java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> > at java.net.ServerSocket.accept(ServerSocket.java:430)
>> > at
>> >
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>> > at
>> >
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>> > at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000
>> waiting on
>> > condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
>> > condition [109fff000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
>> daemon
>> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
>> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>> >
>> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
>> > condition [109cf6000]
>> > java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for <7c1950890> (a
>> > java.util.concurrent.SynchronousQueue$TransferStack)
>> > at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
>> > runnable [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
>> runnable
>> > [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
>> > nid=0x108c78000 waiting on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
>> Object.wait()
>> > [108a6e000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> >
>> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
>> > Object.wait() [10896b000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
>> > at java.lang.Object.wait(Object.java:485)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
>> > [1008e9000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
>> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>> >
>> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
>> > nid=0x103ced000 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
>> > nid=0x103df0000 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
>> > nid=0x103ef3000 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
>> > nid=0x103ff6000 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000
>> nid=0x1084e7000
>> > runnable
>> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
>> waiting
>> > on condition
>> >
>> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
>> runnable
>> > JNI global references: 1457
>> >
>> >
>> >
>> > Best Regards,
>> > Stone
>> >
>> >
>> >
>> >
>> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>> >>
>> >> I got the same issue today. map tasks finished quickly but reduce is
>> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>> >>
>> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
>> 600
>> >> seconds. Killing!
>> >>
>> >>
>> >>
>> >> logs for the reducer :
>> >>
>> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>> >>
>> >>
>> >>
>> >> stdout logs
>> >>
>> >>
>> >> stderr logs
>> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> >> SCDynamicStore
>> >>
>> >>
>> >> syslog logs
>> >> 2012-08-31 01:13:06,421 INFO
>> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> already
>> >> set up for Hadoop, not re-installing.
>> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> >> Unable to load native-hadoop library for your platform... using
>> builtin-java
>> >> classes where applicable
>> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> >> ResourceCalculatorPlugin : null
>> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> ShuffleRamManager: MemoryLimit=144965632,
>> MaxSingleShuffleLimit=36241408
>> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> merging in
>> >> memory files
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> polling Map
>> >> Completion Events
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 0 is
>> >> already in progress
>> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >>
>> >> I can successfully run the wordcount example on my old macbook (os x
>> 10.6)
>> >> previously.
>> >>
>> >> Any suggestions ?
>> >>
>> >> Best Regards,
>> >> Stone
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>> >>>
>> >>>
>> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> >>> but it cannot compute the reduce.
>> >>>
>> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> >>> job_201208130857_0001
>> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>>
>> >>> Here is what I get when I try to see the tasklog using the links
>> given in
>> >>> the output
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> --->
>> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info
>> from
>> >>> SCDynamicStore
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> --->
>> >>>
>> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> >>> https://issues.apache.org/jira/browse/HADOOP-7489
>> >>>
>> >>> Also this error of Unable to load realm info from SCDynamicStore does
>> not
>> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>> >>>
>> >>> I am also attaching a zipped copy of my logs
>> >>>
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Subho.
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
On Fri, Aug 31, 2012 at 3:28 PM, Stone <st...@gmail.com> wrote:
> Fixed this weird issue by change hostname to localhost.
>
> $ hostname
> Stones-MacBook-Air.local
>
> when hostname is Stones-MacBook-Air.local
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
10.18.101.141
> $ scutil --set HostName localhost
> $ hostname
> localhost
>
> when hostname is localhost
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
127.0.0.1
> After changing the hostname restart hadoop. then everything works.
>
> Any ideas about the root cause ?
>
>
> On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Stone,
>>
>> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
>> IPv6?
>>
>> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
>> > FYI : the jstack output for the hanged job:
>> >
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
>> > mode):
>> >
>> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
>> runnable
>> > [109b8f000]
>> > java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> > at java.net.ServerSocket.accept(ServerSocket.java:430)
>> > at
>> >
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>> > at
>> >
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>> > at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000
>> waiting on
>> > condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
>> > condition [109fff000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
>> daemon
>> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
>> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>> >
>> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
>> > condition [109cf6000]
>> > java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for <7c1950890> (a
>> > java.util.concurrent.SynchronousQueue$TransferStack)
>> > at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
>> > runnable [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
>> runnable
>> > [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
>> > nid=0x108c78000 waiting on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
>> Object.wait()
>> > [108a6e000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> >
>> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
>> > Object.wait() [10896b000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
>> > at java.lang.Object.wait(Object.java:485)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
>> > [1008e9000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
>> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>> >
>> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
>> > nid=0x103ced000 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
>> > nid=0x103df0000 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
>> > nid=0x103ef3000 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
>> > nid=0x103ff6000 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000
>> nid=0x1084e7000
>> > runnable
>> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
>> waiting
>> > on condition
>> >
>> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
>> runnable
>> > JNI global references: 1457
>> >
>> >
>> >
>> > Best Regards,
>> > Stone
>> >
>> >
>> >
>> >
>> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>> >>
>> >> I got the same issue today. map tasks finished quickly but reduce is
>> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>> >>
>> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
>> 600
>> >> seconds. Killing!
>> >>
>> >>
>> >>
>> >> logs for the reducer :
>> >>
>> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>> >>
>> >>
>> >>
>> >> stdout logs
>> >>
>> >>
>> >> stderr logs
>> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> >> SCDynamicStore
>> >>
>> >>
>> >> syslog logs
>> >> 2012-08-31 01:13:06,421 INFO
>> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> already
>> >> set up for Hadoop, not re-installing.
>> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> >> Unable to load native-hadoop library for your platform... using
>> builtin-java
>> >> classes where applicable
>> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> >> ResourceCalculatorPlugin : null
>> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> ShuffleRamManager: MemoryLimit=144965632,
>> MaxSingleShuffleLimit=36241408
>> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> merging in
>> >> memory files
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> polling Map
>> >> Completion Events
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 0 is
>> >> already in progress
>> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >>
>> >> I can successfully run the wordcount example on my old macbook (os x
>> 10.6)
>> >> previously.
>> >>
>> >> Any suggestions ?
>> >>
>> >> Best Regards,
>> >> Stone
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>> >>>
>> >>>
>> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> >>> but it cannot compute the reduce.
>> >>>
>> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> >>> job_201208130857_0001
>> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>>
>> >>> Here is what I get when I try to see the tasklog using the links
>> given in
>> >>> the output
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> --->
>> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info
>> from
>> >>> SCDynamicStore
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> --->
>> >>>
>> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> >>> https://issues.apache.org/jira/browse/HADOOP-7489
>> >>>
>> >>> Also this error of Unable to load realm info from SCDynamicStore does
>> not
>> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>> >>>
>> >>> I am also attaching a zipped copy of my logs
>> >>>
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Subho.
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
On Fri, Aug 31, 2012 at 3:28 PM, Stone <st...@gmail.com> wrote:
> Fixed this weird issue by change hostname to localhost.
>
> $ hostname
> Stones-MacBook-Air.local
>
> when hostname is Stones-MacBook-Air.local
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
10.18.101.141
> $ scutil --set HostName localhost
> $ hostname
> localhost
>
> when hostname is localhost
scala> println(java.net.InetAddress.getLocalHost().getCanonicalHostName())
127.0.0.1
> After changing the hostname restart hadoop. then everything works.
>
> Any ideas about the root cause ?
>
>
> On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Stone,
>>
>> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
>> IPv6?
>>
>> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
>> > FYI : the jstack output for the hanged job:
>> >
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
>> > mode):
>> >
>> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
>> runnable
>> > [109b8f000]
>> > java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> > at java.net.ServerSocket.accept(ServerSocket.java:430)
>> > at
>> >
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>> > at
>> >
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>> > at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000
>> waiting on
>> > condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
>> > condition [109fff000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
>> daemon
>> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
>> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>> >
>> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
>> > condition [109cf6000]
>> > java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for <7c1950890> (a
>> > java.util.concurrent.SynchronousQueue$TransferStack)
>> > at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>> > at
>> >
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> > at java.lang.Thread.run(Thread.java:680)
>> >
>> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
>> > runnable [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
>> waiting
>> > on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
>> runnable
>> > [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
>> > nid=0x108c78000 waiting on condition [00000000]
>> > java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
>> Object.wait()
>> > [108a6e000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> >
>> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
>> > Object.wait() [10896b000]
>> > java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
>> > at java.lang.Object.wait(Object.java:485)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
>> > [1008e9000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
>> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>> >
>> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
>> > nid=0x103ced000 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
>> > nid=0x103df0000 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
>> > nid=0x103ef3000 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
>> > nid=0x103ff6000 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000
>> nid=0x1084e7000
>> > runnable
>> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
>> waiting
>> > on condition
>> >
>> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
>> runnable
>> > JNI global references: 1457
>> >
>> >
>> >
>> > Best Regards,
>> > Stone
>> >
>> >
>> >
>> >
>> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>> >>
>> >> I got the same issue today. map tasks finished quickly but reduce is
>> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>> >>
>> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
>> 600
>> >> seconds. Killing!
>> >>
>> >>
>> >>
>> >> logs for the reducer :
>> >>
>> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>> >>
>> >>
>> >>
>> >> stdout logs
>> >>
>> >>
>> >> stderr logs
>> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> >> SCDynamicStore
>> >>
>> >>
>> >> syslog logs
>> >> 2012-08-31 01:13:06,421 INFO
>> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> already
>> >> set up for Hadoop, not re-installing.
>> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> >> Unable to load native-hadoop library for your platform... using
>> builtin-java
>> >> classes where applicable
>> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> >> ResourceCalculatorPlugin : null
>> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> ShuffleRamManager: MemoryLimit=144965632,
>> MaxSingleShuffleLimit=36241408
>> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> >> on-disk files
>> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> merging in
>> >> memory files
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for
>> polling Map
>> >> Completion Events
>> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 0 is
>> >> already in progress
>> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s)
>> where 1 is
>> >> already in progress
>> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
>> and0
>> >> dup hosts)
>> >>
>> >> I can successfully run the wordcount example on my old macbook (os x
>> 10.6)
>> >> previously.
>> >>
>> >> Any suggestions ?
>> >>
>> >> Best Regards,
>> >> Stone
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>> >>>
>> >>>
>> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> >>> but it cannot compute the reduce.
>> >>>
>> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> >>> job_201208130857_0001
>> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
>> outputServer
>> >>> returned HTTP response code: 403 for URL:
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> >>> Too many fetch-failures
>> >>>
>> >>> Here is what I get when I try to see the tasklog using the links
>> given in
>> >>> the output
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> >>> --->
>> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info
>> from
>> >>> SCDynamicStore
>> >>>
>> >>>
>> >>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> >>> --->
>> >>>
>> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> >>> https://issues.apache.org/jira/browse/HADOOP-7489
>> >>>
>> >>> Also this error of Unable to load realm info from SCDynamicStore does
>> not
>> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>> >>>
>> >>> I am also attaching a zipped copy of my logs
>> >>>
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Subho.
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
Fixed this weird issue by change hostname to localhost.
$ hostname
Stones-MacBook-Air.local
$ scutil --set HostName localhost
$ hostname
localhost
After changing the hostname restart hadoop. then everything works.
Any ideas about the root cause ?
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
Fixed this weird issue by change hostname to localhost.
$ hostname
Stones-MacBook-Air.local
$ scutil --set HostName localhost
$ hostname
localhost
After changing the hostname restart hadoop. then everything works.
Any ideas about the root cause ?
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
Fixed this weird issue by change hostname to localhost.
$ hostname
Stones-MacBook-Air.local
$ scutil --set HostName localhost
$ hostname
localhost
After changing the hostname restart hadoop. then everything works.
Any ideas about the root cause ?
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
No. the content in hadoop-env.sh is like the following:
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK-Djava.security.krb5.kdc=kdc0.ox.ac.uk:
kdc1.ox.ac.uk"
Best Regards,
Stone
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
No. the content in hadoop-env.sh is like the following:
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK-Djava.security.krb5.kdc=kdc0.ox.ac.uk:
kdc1.ox.ac.uk"
Best Regards,
Stone
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
Fixed this weird issue by change hostname to localhost.
$ hostname
Stones-MacBook-Air.local
$ scutil --set HostName localhost
$ hostname
localhost
After changing the hostname restart hadoop. then everything works.
Any ideas about the root cause ?
On Fri, Aug 31, 2012 at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Stone,
>
> Do you have any flags in hadoop-env.sh indicating preference of IPv4 over
> IPv6?
>
> On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> > FYI : the jstack output for the hanged job:
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> > mode):
> >
> > "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000
> runnable
> > [109b8f000]
> > java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > at
> >
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> > at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting
> on
> > condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> > condition [109fff000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone"
> daemon
> > prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> > - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
> >
> > "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> > condition [109cf6000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <7c1950890> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> > runnable [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000
> waiting
> > on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000
> runnable
> > [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> > nid=0x108c78000 waiting on condition [00000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in
> Object.wait()
> > [108a6e000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> > "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> > Object.wait() [10896b000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <7c1950860> (a java.lang.ref.Reference$Lock)
> >
> > "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> > [1008e9000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> >
> > "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> > nid=0x103ced000 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> > nid=0x103df0000 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> > nid=0x103ef3000 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> > nid=0x103ff6000 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000
> waiting
> > on condition
> >
> > "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
> runnable
> > JNI global references: 1457
> >
> >
> >
> > Best Regards,
> > Stone
> >
> >
> >
> >
> > On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> >>
> >> I got the same issue today. map tasks finished quickly but reduce is
> >> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
> >>
> >> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> >> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> >> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> >> attempt_201208310112_0001_r_000000_0, Status : FAILED
> >> Task attempt_201208310112_0001_r_000000_0 failed to report status for
> 600
> >> seconds. Killing!
> >>
> >>
> >>
> >> logs for the reducer :
> >>
> >> Task Logs: 'attempt_201208310112_0001_r_000000_0'
> >>
> >>
> >>
> >> stdout logs
> >>
> >>
> >> stderr logs
> >> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> >> SCDynamicStore
> >>
> >>
> >> syslog logs
> >> 2012-08-31 01:13:06,421 INFO
> >> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> already
> >> set up for Hadoop, not re-installing.
> >> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> >> Unable to load native-hadoop library for your platform... using
> builtin-java
> >> classes where applicable
> >> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> >> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> >> ResourceCalculatorPlugin : null
> >> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> >> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> >> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> >> on-disk files
> >> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> in
> >> memory files
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling
> Map
> >> Completion Events
> >> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 0 is
> >> already in progress
> >> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where
> 1 is
> >> already in progress
> >> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> >> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts
> and0
> >> dup hosts)
> >>
> >> I can successfully run the wordcount example on my old macbook (os x
> 10.6)
> >> previously.
> >>
> >> Any suggestions ?
> >>
> >> Best Regards,
> >> Stone
> >>
> >>
> >>
> >>
> >> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
> >>>
> >>>
> >>> When running hadoop on pseudo-distributed mode, the map seems to work,
> >>> but it cannot compute the reduce.
> >>>
> >>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
> >>> job_201208130857_0001
> >>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> >>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> >>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> >>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> >>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> >>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> >>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> >>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> >>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> >>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000000_0, Status : FAILED
> >>> Too many fetch-failures
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task
> outputServer
> >>> returned HTTP response code: 403 for URL:
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> >>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> >>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> >>> attempt_201208130857_0001_m_000001_0, Status : FAILED
> >>> Too many fetch-failures
> >>>
> >>> Here is what I get when I try to see the tasklog using the links given
> in
> >>> the output
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> >>> --->
> >>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> >>> SCDynamicStore
> >>>
> >>>
> >>>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> >>> --->
> >>>
> >>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> >>> https://issues.apache.org/jira/browse/HADOOP-7489
> >>>
> >>> Also this error of Unable to load realm info from SCDynamicStore does
> not
> >>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
> >>>
> >>> I am also attaching a zipped copy of my logs
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Subho.
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Harsh J <ha...@cloudera.com>.
Hi Stone,
Do you have any flags in hadoop-env.sh indicating preference of IPv4 over IPv6?
On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> FYI : the jstack output for the hanged job:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> mode):
>
> "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
> [109b8f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:680)
>
> "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
> condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> condition [109fff000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> at java.lang.Thread.run(Thread.java:680)
>
> "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
> prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>
> "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> condition [109cf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <7c1950890> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:680)
>
> "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> runnable [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
> [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> nid=0x108c78000 waiting on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
> [108a6e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> Object.wait() [10896b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> [1008e9000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
> "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> nid=0x103ced000 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> nid=0x103df0000 runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> nid=0x103ef3000 runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> nid=0x103ff6000 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> runnable
> "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
> on condition
>
> "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000 runnable
> JNI global references: 1457
>
>
>
> Best Regards,
> Stone
>
>
>
>
> On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>>
>> I got the same issue today. map tasks finished quickly but reduce is
>> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>>
>> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
>> seconds. Killing!
>>
>>
>>
>> logs for the reducer :
>>
>> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>>
>>
>>
>> stdout logs
>>
>>
>> stderr logs
>> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> syslog logs
>> 2012-08-31 01:13:06,421 INFO
>> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
>> set up for Hadoop, not re-installing.
>> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> ResourceCalculatorPlugin : null
>> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
>> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
>> memory files
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
>> Completion Events
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0 is
>> already in progress
>> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>>
>> I can successfully run the wordcount example on my old macbook (os x 10.6)
>> previously.
>>
>> Any suggestions ?
>>
>> Best Regards,
>> Stone
>>
>>
>>
>>
>> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
--
Harsh J
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Harsh J <ha...@cloudera.com>.
Hi Stone,
Do you have any flags in hadoop-env.sh indicating preference of IPv4 over IPv6?
On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> FYI : the jstack output for the hanged job:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> mode):
>
> "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
> [109b8f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:680)
>
> "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
> condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> condition [109fff000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> at java.lang.Thread.run(Thread.java:680)
>
> "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
> prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>
> "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> condition [109cf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <7c1950890> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:680)
>
> "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> runnable [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
> [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> nid=0x108c78000 waiting on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
> [108a6e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> Object.wait() [10896b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> [1008e9000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
> "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> nid=0x103ced000 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> nid=0x103df0000 runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> nid=0x103ef3000 runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> nid=0x103ff6000 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> runnable
> "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
> on condition
>
> "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000 runnable
> JNI global references: 1457
>
>
>
> Best Regards,
> Stone
>
>
>
>
> On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>>
>> I got the same issue today. map tasks finished quickly but reduce is
>> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>>
>> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
>> seconds. Killing!
>>
>>
>>
>> logs for the reducer :
>>
>> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>>
>>
>>
>> stdout logs
>>
>>
>> stderr logs
>> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> syslog logs
>> 2012-08-31 01:13:06,421 INFO
>> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
>> set up for Hadoop, not re-installing.
>> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> ResourceCalculatorPlugin : null
>> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
>> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
>> memory files
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
>> Completion Events
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0 is
>> already in progress
>> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>>
>> I can successfully run the wordcount example on my old macbook (os x 10.6)
>> previously.
>>
>> Any suggestions ?
>>
>> Best Regards,
>> Stone
>>
>>
>>
>>
>> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
--
Harsh J
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Harsh J <ha...@cloudera.com>.
Hi Stone,
Do you have any flags in hadoop-env.sh indicating preference of IPv4 over IPv6?
On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> FYI : the jstack output for the hanged job:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> mode):
>
> "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
> [109b8f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:680)
>
> "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
> condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> condition [109fff000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> at java.lang.Thread.run(Thread.java:680)
>
> "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
> prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>
> "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> condition [109cf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <7c1950890> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:680)
>
> "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> runnable [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
> [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> nid=0x108c78000 waiting on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
> [108a6e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> Object.wait() [10896b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> [1008e9000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
> "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> nid=0x103ced000 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> nid=0x103df0000 runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> nid=0x103ef3000 runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> nid=0x103ff6000 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> runnable
> "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
> on condition
>
> "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000 runnable
> JNI global references: 1457
>
>
>
> Best Regards,
> Stone
>
>
>
>
> On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>>
>> I got the same issue today. map tasks finished quickly but reduce is
>> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>>
>> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
>> seconds. Killing!
>>
>>
>>
>> logs for the reducer :
>>
>> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>>
>>
>>
>> stdout logs
>>
>>
>> stderr logs
>> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> syslog logs
>> 2012-08-31 01:13:06,421 INFO
>> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
>> set up for Hadoop, not re-installing.
>> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> ResourceCalculatorPlugin : null
>> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
>> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
>> memory files
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
>> Completion Events
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0 is
>> already in progress
>> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>>
>> I can successfully run the wordcount example on my old macbook (os x 10.6)
>> previously.
>>
>> Any suggestions ?
>>
>> Best Regards,
>> Stone
>>
>>
>>
>>
>> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
--
Harsh J
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Harsh J <ha...@cloudera.com>.
Hi Stone,
Do you have any flags in hadoop-env.sh indicating preference of IPv4 over IPv6?
On Thu, Aug 30, 2012 at 11:26 PM, Stone <st...@gmail.com> wrote:
> FYI : the jstack output for the hanged job:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
> mode):
>
> "RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
> [109b8f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <7bd6a00a8> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:680)
>
> "Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
> condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
> condition [109fff000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
> at java.lang.Thread.run(Thread.java:680)
>
> "IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
> prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
> - locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
>
> "sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
> condition [109cf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <7c1950890> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:680)
>
> "Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
> runnable [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
> on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
> [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
> nid=0x108c78000 waiting on condition [00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
> [108a6e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
> Object.wait() [10896b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <7c1950860> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
> [1008e9000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
> "VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
> nid=0x103ced000 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
> nid=0x103df0000 runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
> nid=0x103ef3000 runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
> nid=0x103ff6000 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
> runnable
> "VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
> on condition
>
> "Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000 runnable
> JNI global references: 1457
>
>
>
> Best Regards,
> Stone
>
>
>
>
> On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
>>
>> I got the same issue today. map tasks finished quickly but reduce is
>> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>>
>> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
>> attempt_201208310112_0001_r_000000_0, Status : FAILED
>> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
>> seconds. Killing!
>>
>>
>>
>> logs for the reducer :
>>
>> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>>
>>
>>
>> stdout logs
>>
>>
>> stderr logs
>> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> syslog logs
>> 2012-08-31 01:13:06,421 INFO
>> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
>> set up for Hadoop, not re-installing.
>> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
>> ResourceCalculatorPlugin : null
>> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
>> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
>> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
>> on-disk files
>> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
>> memory files
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
>> Completion Events
>> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0 is
>> already in progress
>> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1 is
>> already in progress
>> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
>> dup hosts)
>>
>> I can successfully run the wordcount example on my old macbook (os x 10.6)
>> previously.
>>
>> Any suggestions ?
>>
>> Best Regards,
>> Stone
>>
>>
>>
>>
>> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>>
>>>
>>> When running hadoop on pseudo-distributed mode, the map seems to work,
>>> but it cannot compute the reduce.
>>>
>>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>>> job_201208130857_0001
>>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>>> Too many fetch-failures
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>>> returned HTTP response code: 403 for URL:
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>>> Too many fetch-failures
>>>
>>> Here is what I get when I try to see the tasklog using the links given in
>>> the output
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>>> --->
>>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>>> SCDynamicStore
>>>
>>>
>>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>>> --->
>>>
>>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>>> https://issues.apache.org/jira/browse/HADOOP-7489
>>>
>>> Also this error of Unable to load realm info from SCDynamicStore does not
>>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>>
>>> I am also attaching a zipped copy of my logs
>>>
>>>
>>> Cheers,
>>>
>>> Subho.
>>
>>
>
--
Harsh J
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
FYI : the jstack output for the hanged job:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
mode):
"RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
[109b8f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <7bd6a00a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:680)
"Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
condition [00000000]
java.lang.Thread.State: RUNNABLE
"LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
condition [109fff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
at java.lang.Thread.run(Thread.java:680)
"IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
- locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
"sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
condition [109cf6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <7c1950890> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
[00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
nid=0x108c78000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
[108a6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
Object.wait() [10896b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7c1950860> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
[1008e9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
"VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
nid=0x103ced000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
nid=0x103df0000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
nid=0x103ef3000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
nid=0x103ff6000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
runnable
"VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
on condition
"Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
runnable
JNI global references: 1457
Best Regards,
Stone
On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> I got the same issue today. map tasks finished quickly but reduce is
> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>
> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> attempt_201208310112_0001_r_000000_0, Status : FAILED
> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
> seconds. Killing!
>
>
>
> logs for the reducer :
>
> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>
>
>
> stdout logs
>
>
> stderr logs
> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> SCDynamicStore
>
>
> syslog logs
> 2012-08-31 01:13:06,421 INFO
> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
> set up for Hadoop, not re-installing.
> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin : null
> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
> memory files
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
> Completion Events
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
> is already in progress
> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
>
> I can successfully run the wordcount example on my old macbook (os x 10.6)
> previously.
>
> Any suggestions ?
>
> Best Regards,
> Stone
>
>
>
>
> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
FYI : the jstack output for the hanged job:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
mode):
"RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
[109b8f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <7bd6a00a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:680)
"Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
condition [00000000]
java.lang.Thread.State: RUNNABLE
"LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
condition [109fff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
at java.lang.Thread.run(Thread.java:680)
"IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
- locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
"sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
condition [109cf6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <7c1950890> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
[00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
nid=0x108c78000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
[108a6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
Object.wait() [10896b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7c1950860> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
[1008e9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
"VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
nid=0x103ced000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
nid=0x103df0000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
nid=0x103ef3000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
nid=0x103ff6000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
runnable
"VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
on condition
"Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
runnable
JNI global references: 1457
Best Regards,
Stone
On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> I got the same issue today. map tasks finished quickly but reduce is
> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>
> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> attempt_201208310112_0001_r_000000_0, Status : FAILED
> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
> seconds. Killing!
>
>
>
> logs for the reducer :
>
> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>
>
>
> stdout logs
>
>
> stderr logs
> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> SCDynamicStore
>
>
> syslog logs
> 2012-08-31 01:13:06,421 INFO
> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
> set up for Hadoop, not re-installing.
> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin : null
> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
> memory files
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
> Completion Events
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
> is already in progress
> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
>
> I can successfully run the wordcount example on my old macbook (os x 10.6)
> previously.
>
> Any suggestions ?
>
> Best Regards,
> Stone
>
>
>
>
> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
FYI : the jstack output for the hanged job:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
mode):
"RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
[109b8f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <7bd6a00a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:680)
"Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
condition [00000000]
java.lang.Thread.State: RUNNABLE
"LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
condition [109fff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
at java.lang.Thread.run(Thread.java:680)
"IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
- locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
"sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
condition [109cf6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <7c1950890> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
[00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
nid=0x108c78000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
[108a6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
Object.wait() [10896b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7c1950860> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
[1008e9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
"VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
nid=0x103ced000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
nid=0x103df0000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
nid=0x103ef3000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
nid=0x103ff6000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
runnable
"VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
on condition
"Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
runnable
JNI global references: 1457
Best Regards,
Stone
On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> I got the same issue today. map tasks finished quickly but reduce is
> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>
> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> attempt_201208310112_0001_r_000000_0, Status : FAILED
> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
> seconds. Killing!
>
>
>
> logs for the reducer :
>
> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>
>
>
> stdout logs
>
>
> stderr logs
> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> SCDynamicStore
>
>
> syslog logs
> 2012-08-31 01:13:06,421 INFO
> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
> set up for Hadoop, not re-installing.
> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin : null
> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
> memory files
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
> Completion Events
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
> is already in progress
> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
>
> I can successfully run the wordcount example on my old macbook (os x 10.6)
> previously.
>
> Any suggestions ?
>
> Best Regards,
> Stone
>
>
>
>
> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
FYI : the jstack output for the hanged job:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed
mode):
"RMI TCP Accept-0" daemon prio=9 tid=7f833f1ba800 nid=0x109b90000 runnable
[109b8f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <7bd6a00a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:680)
"Attach Listener" daemon prio=9 tid=7f833d000800 nid=0x108b72000 waiting on
condition [00000000]
java.lang.Thread.State: RUNNABLE
"LeaseChecker" daemon prio=5 tid=7f833f30d000 nid=0x10a000000 waiting on
condition [109fff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
at java.lang.Thread.run(Thread.java:680)
"IPC Client (47) connection to localhost/127.0.0.1:9001 from stone" daemon
prio=5 tid=7f833d0de000 nid=0x109dfa000 in Object.wait() [109df9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
- locked <7bd6a02e0> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
"sendParams-0" daemon prio=5 tid=7f833e1cb000 nid=0x109cf7000 waiting on
condition [109cf6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <7c1950890> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=7f833e021800 nid=0x109084000
runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7f833e021000 nid=0x108f81000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7f833e020000 nid=0x108e7e000 waiting
on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7f833e01f800 nid=0x108d7b000 runnable
[00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f833e01e800
nid=0x108c78000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7f833e00f000 nid=0x108a6f000 in Object.wait()
[108a6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7c19c1550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7f833e00e000 nid=0x10896c000 in
Object.wait() [10896b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7c1950860> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7c1950860> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7f833f000800 nid=0x1008eb000 waiting on condition
[1008e9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1316)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:509)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
"VM Thread" prio=9 tid=7f833e009800 nid=0x108869000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f833f002000
nid=0x103ced000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f833f002800
nid=0x103df0000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7f833f003000
nid=0x103ef3000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7f833f004000
nid=0x103ff6000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7f833f07f000 nid=0x1084e7000
runnable
"VM Periodic Task Thread" prio=10 tid=7f833e033800 nid=0x109187000 waiting
on condition
"Exception Catcher Thread" prio=10 tid=7f833f001800 nid=0x100b16000
runnable
JNI global references: 1457
Best Regards,
Stone
On Fri, Aug 31, 2012 at 1:24 AM, Stone <st...@gmail.com> wrote:
> I got the same issue today. map tasks finished quickly but reduce is
> always 0%. I am also running Mac OS X 10.8. (cdh3u4)
>
> 12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
> attempt_201208310112_0001_r_000000_0, Status : FAILED
> Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
> seconds. Killing!
>
>
>
> logs for the reducer :
>
> Task Logs: 'attempt_201208310112_0001_r_000000_0'
>
>
>
> stdout logs
>
>
> stderr logs
> 2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
> SCDynamicStore
>
>
> syslog logs
> 2012-08-31 01:13:06,421 INFO
> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
> set up for Hadoop, not re-installing.
> 2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
> 2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin : null
> 2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
> ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
> 2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
> on-disk files
> 2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
> memory files
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
> Completion Events
> 2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
> is already in progress
> 2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
> is already in progress
> 2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
>
> I can successfully run the wordcount example on my old macbook (os x 10.6)
> previously.
>
> Any suggestions ?
>
> Best Regards,
> Stone
>
>
>
>
> On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
>
>> Hello,
>>
>> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>>
>>
>> When running hadoop on pseudo-distributed mode, the map seems to work,
>> but it cannot compute the reduce.
>>
>> 12/08/13 08:58:12 INFO mapred.JobClient: Running job:
>> job_201208130857_0001
>> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
>> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
>> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
>> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
>> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
>> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
>> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
>> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
>> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
>> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000000_0, Status : FAILED
>> Too many fetch-failures
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
>> returned HTTP response code: 403 for URL:
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
>> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
>> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
>> attempt_201208130857_0001_m_000001_0, Status : FAILED
>> Too many fetch-failures
>>
>> Here is what I get when I try to see the tasklog using the links given in
>> the output
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>> --->
>> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
>> SCDynamicStore
>>
>>
>> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>> --->
>>
>> I have changed my hadoop-env.sh acoording to Mathew Buckett in
>> https://issues.apache.org/jira/browse/HADOOP-7489
>>
>> Also this error of Unable to load realm info from SCDynamicStore does not
>> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>>
>> I am also attaching a zipped copy of my logs
>>
>>
>> Cheers,
>>
>> Subho.
>>
>
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
I got the same issue today. map tasks finished quickly but reduce is always
0%. I am also running Mac OS X 10.8. (cdh3u4)
12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
attempt_201208310112_0001_r_000000_0, Status : FAILED
Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
seconds. Killing!
logs for the reducer :
Task Logs: 'attempt_201208310112_0001_r_000000_0'
stdout logs
stderr logs
2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
SCDynamicStore
syslog logs
2012-08-31 01:13:06,421 INFO
org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin : null
2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
memory files
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
Completion Events
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
is already in progress
2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
I can successfully run the wordcount example on my old macbook (os x 10.6)
previously.
Any suggestions ?
Best Regards,
Stone
On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Pseudo-distributed mode is good for developing and testing hadoop
code. But instead of experimenting with hadoop on your mac, I would go
for hadoop on EC2. With starcluster http://web.mit.edu/star/cluster/
it takes just a single command to start hadoop. You also get a fixed
environment.
-Håvard
On Mon, Aug 13, 2012 at 6:21 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Subho,
Please check the permission of mapred.local.dir is. This is the
place where map outputs are store. Reduce phase is sending the read
requests but this directory is not accessible. As a result 403 is thrown.
Regards,
Mohammad Tariq
On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Pseudo-distributed mode is good for developing and testing hadoop
code. But instead of experimenting with hadoop on your mac, I would go
for hadoop on EC2. With starcluster http://web.mit.edu/star/cluster/
it takes just a single command to start hadoop. You also get a fixed
environment.
-Håvard
On Mon, Aug 13, 2012 at 6:21 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
I got the same issue today. map tasks finished quickly but reduce is always
0%. I am also running Mac OS X 10.8. (cdh3u4)
12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
attempt_201208310112_0001_r_000000_0, Status : FAILED
Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
seconds. Killing!
logs for the reducer :
Task Logs: 'attempt_201208310112_0001_r_000000_0'
stdout logs
stderr logs
2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
SCDynamicStore
syslog logs
2012-08-31 01:13:06,421 INFO
org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin : null
2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
memory files
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
Completion Events
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
is already in progress
2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
I can successfully run the wordcount example on my old macbook (os x 10.6)
previously.
Any suggestions ?
Best Regards,
Stone
On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Pseudo-distributed mode is good for developing and testing hadoop
code. But instead of experimenting with hadoop on your mac, I would go
for hadoop on EC2. With starcluster http://web.mit.edu/star/cluster/
it takes just a single command to start hadoop. You also get a fixed
environment.
-Håvard
On Mon, Aug 13, 2012 at 6:21 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
I got the same issue today. map tasks finished quickly but reduce is always
0%. I am also running Mac OS X 10.8. (cdh3u4)
12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
attempt_201208310112_0001_r_000000_0, Status : FAILED
Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
seconds. Killing!
logs for the reducer :
Task Logs: 'attempt_201208310112_0001_r_000000_0'
stdout logs
stderr logs
2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
SCDynamicStore
syslog logs
2012-08-31 01:13:06,421 INFO
org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin : null
2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
memory files
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
Completion Events
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
is already in progress
2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
I can successfully run the wordcount example on my old macbook (os x 10.6)
previously.
Any suggestions ?
Best Regards,
Stone
On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Subho,
Please check the permission of mapred.local.dir is. This is the
place where map outputs are store. Reduce phase is sending the read
requests but this directory is not accessible. As a result 403 is thrown.
Regards,
Mohammad Tariq
On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Subho,
Please check the permission of mapred.local.dir is. This is the
place where map outputs are store. Reduce phase is sending the read
requests but this directory is not accessible. As a result 403 is thrown.
Regards,
Mohammad Tariq
On Mon, Aug 13, 2012 at 9:51 AM, Subho Banerjee <su...@gmail.com> wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>
Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Posted by Stone <st...@gmail.com>.
I got the same issue today. map tasks finished quickly but reduce is always
0%. I am also running Mac OS X 10.8. (cdh3u4)
12/08/31 01:13:03 INFO mapred.JobClient: map 0% reduce 0%
12/08/31 01:13:07 INFO mapred.JobClient: map 100% reduce 0%
12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
attempt_201208310112_0001_r_000000_0, Status : FAILED
Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
seconds. Killing!
logs for the reducer :
Task Logs: 'attempt_201208310112_0001_r_000000_0'
stdout logs
stderr logs
2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
SCDynamicStore
syslog logs
2012-08-31 01:13:06,421 INFO
org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin : null
2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
memory files
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
Completion Events
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
is already in progress
2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
I can successfully run the wordcount example on my old macbook (os x 10.6)
previously.
Any suggestions ?
Best Regards,
Stone
On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <su...@gmail.com>wrote:
> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>