You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by evelina dumitrescu <ev...@gmail.com> on 2017/04/21 05:01:18 UTC
Noob question about Hadoop job that writes output to HBase
Hi,
I am new to Hadoop and Hbase.
I was trying to make a small proof-of-concept Hadoop map reduce job that
reads the data from HDFS and stores the output in Hbase.
I did the setup as presented in this tutorial [1].
Here is the pseudocode from the map reduce code [2].
The problem is that I am unable to contact Hbase from the Hadoop job and
the job gets stuck.
Here are the logs from syslog [3], stderr [4] and console [5].
How should I correctly setup HbaseConfiguration ?
I couldn't find any example online that worked and it's hard for a beginner
to debug the issue.
Any help would be appreciated.
Thank you,
Evelina
[1]
http://www.bogotobogo.com/Hadoop/BigData_hadoop_HBase_Pseudo_Distributed.php
[2]
https://pastebin.com/hUDAMMes
[3]
https://pastebin.com/XxmWAUTf
[4]
https://pastebin.com/fYUYw4Cv
[5]
https://pastebin.com/YJ1hERDe
Re: Noob question about Hadoop job that writes output to HBase
Posted by Sidharth Kumar <si...@gmail.com>.
I guess even aggregated log will have error which can be collected by using
yarn logs -applicationId <application Id> > <application Id>.log
Sidharth
Mob: +91 8197555599
LinkedIn: www.linkedin.com/in/sidharthkumar2792
On 22-Apr-2017 5:40 PM, "Ravi Prakash" <ra...@gmail.com> wrote:
> Hi Evelina!
>
> You've posted the logs for the MapReduce ApplicationMaster . From this I
> can see the reducer timed out after 600 secs :
> 2017-04-21 00:24:07,747 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1492722585320_0001_r_000000_0: AttemptID:attempt_1492722585320_0001_r_000000_0
> Timed out after 600 secs
>
> To find out why the reducer timed out, you'd have to go look at the logs
> of the reducer. These too are available from the RM page (where you got
> these logs)..... just click a few more links deeper.
>
> HTH
> Ravi
>
> On Fri, Apr 21, 2017 at 1:58 AM, evelina dumitrescu <
> evelina.a.dumitrescu@gmail.com> wrote:
>
>> The Hadoop version that I use is 2.7.1 and the Hbase version is 1.2.5.
>> I can do any operation from the HBase shell.
>>
>>
>> On Fri, Apr 21, 2017 at 8:01 AM, evelina dumitrescu <
>> evelina.a.dumitrescu@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am new to Hadoop and Hbase.
>>> I was trying to make a small proof-of-concept Hadoop map reduce job that
>>> reads the data from HDFS and stores the output in Hbase.
>>> I did the setup as presented in this tutorial [1].
>>> Here is the pseudocode from the map reduce code [2].
>>> The problem is that I am unable to contact Hbase from the Hadoop job and
>>> the job gets stuck.
>>> Here are the logs from syslog [3], stderr [4] and console [5].
>>> How should I correctly setup HbaseConfiguration ?
>>> I couldn't find any example online that worked and it's hard for a
>>> beginner to debug the issue.
>>> Any help would be appreciated.
>>>
>>> Thank you,
>>> Evelina
>>>
>>> [1]
>>> http://www.bogotobogo.com/Hadoop/BigData_hadoop_HBase_Pseudo
>>> _Distributed.php
>>> [2]
>>> https://pastebin.com/hUDAMMes
>>> [3]
>>> https://pastebin.com/XxmWAUTf
>>> [4]
>>> https://pastebin.com/fYUYw4Cv
>>> [5]
>>> https://pastebin.com/YJ1hERDe
>>>
>>
>>
>
Re: Noob question about Hadoop job that writes output to HBase
Posted by Ravi Prakash <ra...@gmail.com>.
Hi Evelina!
You've posted the logs for the MapReduce ApplicationMaster . From this I
can see the reducer timed out after 600 secs :
2017-04-21 00:24:07,747 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
report from attempt_1492722585320_0001_r_000000_0:
AttemptID:attempt_1492722585320_0001_r_000000_0 Timed out after 600 secs
To find out why the reducer timed out, you'd have to go look at the logs of
the reducer. These too are available from the RM page (where you got these
logs)..... just click a few more links deeper.
HTH
Ravi
On Fri, Apr 21, 2017 at 1:58 AM, evelina dumitrescu <
evelina.a.dumitrescu@gmail.com> wrote:
> The Hadoop version that I use is 2.7.1 and the Hbase version is 1.2.5.
> I can do any operation from the HBase shell.
>
>
> On Fri, Apr 21, 2017 at 8:01 AM, evelina dumitrescu <
> evelina.a.dumitrescu@gmail.com> wrote:
>
>> Hi,
>>
>> I am new to Hadoop and Hbase.
>> I was trying to make a small proof-of-concept Hadoop map reduce job that
>> reads the data from HDFS and stores the output in Hbase.
>> I did the setup as presented in this tutorial [1].
>> Here is the pseudocode from the map reduce code [2].
>> The problem is that I am unable to contact Hbase from the Hadoop job and
>> the job gets stuck.
>> Here are the logs from syslog [3], stderr [4] and console [5].
>> How should I correctly setup HbaseConfiguration ?
>> I couldn't find any example online that worked and it's hard for a
>> beginner to debug the issue.
>> Any help would be appreciated.
>>
>> Thank you,
>> Evelina
>>
>> [1]
>> http://www.bogotobogo.com/Hadoop/BigData_hadoop_HBase_Pseudo
>> _Distributed.php
>> [2]
>> https://pastebin.com/hUDAMMes
>> [3]
>> https://pastebin.com/XxmWAUTf
>> [4]
>> https://pastebin.com/fYUYw4Cv
>> [5]
>> https://pastebin.com/YJ1hERDe
>>
>
>
Re: Noob question about Hadoop job that writes output to HBase
Posted by evelina dumitrescu <ev...@gmail.com>.
The Hadoop version that I use is 2.7.1 and the Hbase version is 1.2.5.
I can do any operation from the HBase shell.
On Fri, Apr 21, 2017 at 8:01 AM, evelina dumitrescu <
evelina.a.dumitrescu@gmail.com> wrote:
> Hi,
>
> I am new to Hadoop and Hbase.
> I was trying to make a small proof-of-concept Hadoop map reduce job that
> reads the data from HDFS and stores the output in Hbase.
> I did the setup as presented in this tutorial [1].
> Here is the pseudocode from the map reduce code [2].
> The problem is that I am unable to contact Hbase from the Hadoop job and
> the job gets stuck.
> Here are the logs from syslog [3], stderr [4] and console [5].
> How should I correctly setup HbaseConfiguration ?
> I couldn't find any example online that worked and it's hard for a
> beginner to debug the issue.
> Any help would be appreciated.
>
> Thank you,
> Evelina
>
> [1]
> http://www.bogotobogo.com/Hadoop/BigData_hadoop_HBase_Pseudo
> _Distributed.php
> [2]
> https://pastebin.com/hUDAMMes
> [3]
> https://pastebin.com/XxmWAUTf
> [4]
> https://pastebin.com/fYUYw4Cv
> [5]
> https://pastebin.com/YJ1hERDe
>