You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Ding, Hui" <hu...@sap.com> on 2008/09/20 02:07:56 UTC
Problem writing data into table using M/R
Hi all,
I'm having a problem writing data into hbase table using a m/r program.
I tried to run the sample code by Allen Day (the first one on the wiki
page:http://wiki.apache.org/hadoop/Hbase/MapReduce)
I put some logging statements in the mapper and see that there are the
same number of mappers executed as there are cells to insert. I don't
see any error message etc,
But then I don't see any data inserted after the program terminates!
Meanwhile I was able to do the insertion and scan using the Client API
and it works fine.
Can someone please give me some hint where to look for problem?
Thanks a lot!!!
Re: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by stack <st...@duboce.net>.
Ding, Hui wrote:
> I added some logs and learned that every mapper function actually got
> executed to the end (table commit).
>
>
So, maps completed? Are you using TableReduce? Maybe you need to
instrument this? Where is your job hung up? When it fails, what task
does it report as the problem one? If you look at the task log, any
clues? Does it say timed out because failed report to TaskTracker? Was
it a map or a reduce task?
St.Ack
RE: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by "Ding, Hui" <hu...@sap.com>.
Please see comments below.
-----Original Message-----
From: stack [mailto:stack@duboce.net]
Sent: Tuesday, September 23, 2008 10:18 AM
To: hbase-user@hadoop.apache.org
Subject: Re: [LIKELY JUNK]Re: Problem writing data into table using M/R
Ding, Hui wrote:
> Do you think it is something with contention on the row? I tried to
look
> at the log of the regionserver but there is nothing. Maybe I need to
> enable some setting to see the full log (the logs I saw were
> surprisingly limited)?
>
*Shouldn't* be a problem. Each update will take out a lock while
modifying the cell but should be fine; all mappers should get succeed.
I added some logs and learned that every mapper function actually got
executed to the end (table commit).
For this case, you should probably add log to your task; add log before
and after updates. Try to figure why/how its hanging.
I'm sure there is some way for me to find out what's happening to each
individual mapper, how can I log this information? Can you be a little
more specific? Thanks a lot!
Thanks for persisting.
St.Ack
Re: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by stack <st...@duboce.net>.
Ding, Hui wrote:
> Do you think it is something with contention on the row? I tried to look
> at the log of the regionserver but there is nothing. Maybe I need to
> enable some setting to see the full log (the logs I saw were
> surprisingly limited)?
>
*Shouldn't* be a problem. Each update will take out a lock while
modifying the cell but should be fine; all mappers should get succeed.
For this case, you should probably add log to your task; add log before
and after updates. Try to figure why/how its hanging.
Thanks for persisting.
St.Ack
RE: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by "Ding, Hui" <hu...@sap.com>.
Thanks for your response Stack!
The original program writes to a cell in each of the mapper, there are 4
cells per row so 4 mappers may be accessing a cell together.
I modified that using just one mapper per row and the problem went away
(I was able to insert into the table).
Do you think it is something with contention on the row? I tried to look
at the log of the regionserver but there is nothing. Maybe I need to
enable some setting to see the full log (the logs I saw were
surprisingly limited)?
-----Original Message-----
From: stack [mailto:stack@duboce.net]
Sent: Monday, September 22, 2008 10:18 PM
To: hbase-user@hadoop.apache.org
Subject: Re: [LIKELY JUNK]Re: Problem writing data into table using M/R
Ding, Hui wrote:
> Allen, thanks in advance!
>
> The bulk loading from flatfile to htable one.
> One thing I'm little confused is that the file path I supplied is to
the
> local path not the dfs, is this correct?
> But when I tried to move the file to dfs and use the path there, it
> gives me an error saying file cannot be found.
>
Whats in your hadoop configuration as default filesystem? Is it local
filesystem? Whatever is configured in hadoop-site.xml as your
fs.default.name will be taken as the 'default' filesystem when you run
your MR job (usually).
You might try using a file specification that includes the filesystem
when referring to the file up in hdfs: e.g.
hdfs://HOST:PORT/WHERE/IN/HDFS.
St.Ack
> -----Original Message-----
> From: Allen Day [mailto:allenday@gmail.com]
> Sent: Friday, September 19, 2008 5:15 PM
> To: hbase-user@hadoop.apache.org
> Subject: [LIKELY JUNK]Re: Problem writing data into table using M/R
>
> Hi Ding,
>
> Caveat emptor, I'm also new to HBase and did those examples as part of
> learning the API. But I'll try to help you.
>
> Which example are you using the flatfile -> HTable, or HTable ->
HTable?
>
> -Allen
>
> On Fri, Sep 19, 2008 at 5:07 PM, Ding, Hui <hu...@sap.com> wrote:
>
>> Hi all,
>>
>> I'm having a problem writing data into hbase table using a m/r
>>
> program.
>
>> I tried to run the sample code by Allen Day (the first one on the
wiki
>> page:http://wiki.apache.org/hadoop/Hbase/MapReduce)
>> I put some logging statements in the mapper and see that there are
the
>> same number of mappers executed as there are cells to insert. I
don't
>> see any error message etc,
>> But then I don't see any data inserted after the program terminates!
>>
>> Meanwhile I was able to do the insertion and scan using the Client
API
>> and it works fine.
>>
>> Can someone please give me some hint where to look for problem?
>>
>> Thanks a lot!!!
>>
>>
Re: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by stack <st...@duboce.net>.
Ding, Hui wrote:
> Allen, thanks in advance!
>
> The bulk loading from flatfile to htable one.
> One thing I'm little confused is that the file path I supplied is to the
> local path not the dfs, is this correct?
> But when I tried to move the file to dfs and use the path there, it
> gives me an error saying file cannot be found.
>
Whats in your hadoop configuration as default filesystem? Is it local
filesystem? Whatever is configured in hadoop-site.xml as your
fs.default.name will be taken as the 'default' filesystem when you run
your MR job (usually).
You might try using a file specification that includes the filesystem
when referring to the file up in hdfs: e.g. hdfs://HOST:PORT/WHERE/IN/HDFS.
St.Ack
> -----Original Message-----
> From: Allen Day [mailto:allenday@gmail.com]
> Sent: Friday, September 19, 2008 5:15 PM
> To: hbase-user@hadoop.apache.org
> Subject: [LIKELY JUNK]Re: Problem writing data into table using M/R
>
> Hi Ding,
>
> Caveat emptor, I'm also new to HBase and did those examples as part of
> learning the API. But I'll try to help you.
>
> Which example are you using the flatfile -> HTable, or HTable -> HTable?
>
> -Allen
>
> On Fri, Sep 19, 2008 at 5:07 PM, Ding, Hui <hu...@sap.com> wrote:
>
>> Hi all,
>>
>> I'm having a problem writing data into hbase table using a m/r
>>
> program.
>
>> I tried to run the sample code by Allen Day (the first one on the wiki
>> page:http://wiki.apache.org/hadoop/Hbase/MapReduce)
>> I put some logging statements in the mapper and see that there are the
>> same number of mappers executed as there are cells to insert. I don't
>> see any error message etc,
>> But then I don't see any data inserted after the program terminates!
>>
>> Meanwhile I was able to do the insertion and scan using the Client API
>> and it works fine.
>>
>> Can someone please give me some hint where to look for problem?
>>
>> Thanks a lot!!!
>>
>>
RE: [LIKELY JUNK]Re: Problem writing data into table using M/R
Posted by "Ding, Hui" <hu...@sap.com>.
Allen, thanks in advance!
The bulk loading from flatfile to htable one.
One thing I'm little confused is that the file path I supplied is to the
local path not the dfs, is this correct?
But when I tried to move the file to dfs and use the path there, it
gives me an error saying file cannot be found.
-----Original Message-----
From: Allen Day [mailto:allenday@gmail.com]
Sent: Friday, September 19, 2008 5:15 PM
To: hbase-user@hadoop.apache.org
Subject: [LIKELY JUNK]Re: Problem writing data into table using M/R
Hi Ding,
Caveat emptor, I'm also new to HBase and did those examples as part of
learning the API. But I'll try to help you.
Which example are you using the flatfile -> HTable, or HTable -> HTable?
-Allen
On Fri, Sep 19, 2008 at 5:07 PM, Ding, Hui <hu...@sap.com> wrote:
> Hi all,
>
> I'm having a problem writing data into hbase table using a m/r
program.
>
>
> I tried to run the sample code by Allen Day (the first one on the wiki
> page:http://wiki.apache.org/hadoop/Hbase/MapReduce)
> I put some logging statements in the mapper and see that there are the
> same number of mappers executed as there are cells to insert. I don't
> see any error message etc,
> But then I don't see any data inserted after the program terminates!
>
> Meanwhile I was able to do the insertion and scan using the Client API
> and it works fine.
>
> Can someone please give me some hint where to look for problem?
>
> Thanks a lot!!!
>
Re: Problem writing data into table using M/R
Posted by Allen Day <al...@gmail.com>.
Hi Ding,
Caveat emptor, I'm also new to HBase and did those examples as part of
learning the API. But I'll try to help you.
Which example are you using the flatfile -> HTable, or HTable -> HTable?
-Allen
On Fri, Sep 19, 2008 at 5:07 PM, Ding, Hui <hu...@sap.com> wrote:
> Hi all,
>
> I'm having a problem writing data into hbase table using a m/r program.
>
>
> I tried to run the sample code by Allen Day (the first one on the wiki
> page:http://wiki.apache.org/hadoop/Hbase/MapReduce)
> I put some logging statements in the mapper and see that there are the
> same number of mappers executed as there are cells to insert. I don't
> see any error message etc,
> But then I don't see any data inserted after the program terminates!
>
> Meanwhile I was able to do the insertion and scan using the Client API
> and it works fine.
>
> Can someone please give me some hint where to look for problem?
>
> Thanks a lot!!!
>