You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jeremy Carroll <je...@networkedinsights.com> on 2010/08/12 22:13:33 UTC
Issue with StoreFiles with bulk import.
I'm currently importing some files into HBase and am running into an problem with a large number of store files being created. We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand. When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically. I do not know if this is happening because we are flushing the commits more frequently with smaller files. I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
http://twitpic.com/photos/jeremy_carroll
Re: Issue with StoreFiles with bulk import.
Posted by Stack <st...@duboce.net>.
On Fri, Aug 13, 2010 at 8:00 AM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> To speak to the CompactionQueue being overrun. What we are seeing is that after the dust has settled, and all of the smaller sequence files have been imported, the CompactionQueue is 0. It does not even try to compact. The only way to get the storeFiles down is to run a major compaction. We have hbase.hstore.blockingStoreFiles=15. So why is it not compacting a region with 300 storeFiles?
>
If you don't mind, post a good sample from your regionserver logs --
one where you are seeing the 300 storefiles per region -- and your
hbase-site.xml as well as a dump of your table schema. You can mail
offlist if you would prefer.
St.Ack
________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
> Sent: Friday, August 13, 2010 12:33 AM
> To: user@hbase.apache.org
> Subject: Re: Issue with StoreFiles with bulk import.
>
> On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
> <je...@networkedinsights.com> wrote:
>> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
>
> Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
>
>
>> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
>
>
> So when you mapreduce using these big files as source and insert into
> hbase, its not an issue?
>
>
>> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
>
>
> Why you need to change?
>
>
>> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
>
> Probably. Have you tinkered with hbase default settings in any way?
>
> Perhaps you are getting better parallelism when lots of small files to
> chomp on? More concurrent maps/clients? So rate of upload goes up?
>
>
>> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>>
>
>
> The 90 second timeout is the regionserver timing out against
> zookeeper? Or is it something else?
>
> Storefiles are on the filesystem so what do you mean by the above fear
> of their not being committed to disk?
>
>
>> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>>
>> http://twitpic.com/photos/jeremy_carroll
>>
>
>
> Nice pictures.
>
> 30k storefiles is a good number. They will go up as you are doing a
> bulk load as the compactor is probably overrun. HBase will usually
> catch up though especially after the upload completes.
>
> Do you have compression enabled?
>
> I see regions growing steadily rather than spiking as the comment on
> the graph says. 500 regions ain't too many...
>
> How many servers in your cluster?
>
> St.Ack
>
>
>>
>>
>
RE: Issue with StoreFiles with bulk import.
Posted by Jeremy Carroll <je...@networkedinsights.com>.
To speak to the CompactionQueue being overrun. What we are seeing is that after the dust has settled, and all of the smaller sequence files have been imported, the CompactionQueue is 0. It does not even try to compact. The only way to get the storeFiles down is to run a major compaction. We have hbase.hstore.blockingStoreFiles=15. So why is it not compacting a region with 300 storeFiles?
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Friday, August 13, 2010 12:33 AM
To: user@hbase.apache.org
Subject: Re: Issue with StoreFiles with bulk import.
On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
So when you mapreduce using these big files as source and insert into
hbase, its not an issue?
> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
Why you need to change?
> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
Probably. Have you tinkered with hbase default settings in any way?
Perhaps you are getting better parallelism when lots of small files to
chomp on? More concurrent maps/clients? So rate of upload goes up?
> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>
The 90 second timeout is the regionserver timing out against
zookeeper? Or is it something else?
Storefiles are on the filesystem so what do you mean by the above fear
of their not being committed to disk?
> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>
> http://twitpic.com/photos/jeremy_carroll
>
Nice pictures.
30k storefiles is a good number. They will go up as you are doing a
bulk load as the compactor is probably overrun. HBase will usually
catch up though especially after the upload completes.
Do you have compression enabled?
I see regions growing steadily rather than spiking as the comment on
the graph says. 500 regions ain't too many...
How many servers in your cluster?
St.Ack
>
>
RE: Issue with StoreFiles with bulk import.
Posted by Jeremy Carroll <je...@networkedinsights.com>.
Here is a copy of our current configuration
http://pastebin.com/i2fYtp1E
We turned off major compactions. Other than that I do not believe we modified the compaction operation settings. We have a load average of about 159 regions per server. We have 3 region servers in our Dev environment. The dev environment consists of a zookeeper ensemble (3 servers), a name node / hmaster / jobtracker node (1 server), and region servers / datanodes / tasktrackers (3 servers). Total of 7 servers.
Thanks,
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Friday, August 13, 2010 10:13 AM
To: user@hbase.apache.org
Subject: Re: Issue with StoreFiles with bulk import.
On Fri, Aug 13, 2010 at 6:26 AM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> The main issue is that the stores get very fragmented. I've seen as much as 300 storeFiles for one region during this process.
How many column families and doesn't this number start to do down
after a while because they get compacted together? How many regions
per server do you have?
I'm concerned about performance with that many files to search
through. We are seeing this in the UI for a regions list
(storeFiles=?), also we see this through our JMX monitoring as well.
If there is no issue with performance I'm OK. But as it seems right
now the only way I can decrease the storeFile count is do to a major
compaction.
You are doing manual intervention when you do that... hbase should be
compacting in the background bring down the overall numbers of
storefiles per region.
We only do one major compaction per day to minimize load on the
system. So during the day am I going to see drastically reduced
performance with 30,000 storeFiles versus 800 when it's major
compacted? The 90 second timeout was the flush wait timeout. We are
inserting pretty fast so the flush timeout hits it's 90 second limit
and aborts.
Have you changed other configurations that disable the natural hbase
compacting facility?
St.Ack
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
> Sent: Friday, August 13, 2010 12:33 AM
> To: user@hbase.apache.org
> Subject: Re: Issue with StoreFiles with bulk import.
>
> On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
> <je...@networkedinsights.com> wrote:
>> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
>
> Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
>
>
>> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
>
>
> So when you mapreduce using these big files as source and insert into
> hbase, its not an issue?
>
>
>> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
>
>
> Why you need to change?
>
>
>> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
>
> Probably. Have you tinkered with hbase default settings in any way?
>
> Perhaps you are getting better parallelism when lots of small files to
> chomp on? More concurrent maps/clients? So rate of upload goes up?
>
>
>> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>>
>
>
> The 90 second timeout is the regionserver timing out against
> zookeeper? Or is it something else?
>
> Storefiles are on the filesystem so what do you mean by the above fear
> of their not being committed to disk?
>
>
>> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>>
>> http://twitpic.com/photos/jeremy_carroll
>>
>
>
> Nice pictures.
>
> 30k storefiles is a good number. They will go up as you are doing a
> bulk load as the compactor is probably overrun. HBase will usually
> catch up though especially after the upload completes.
>
> Do you have compression enabled?
>
> I see regions growing steadily rather than spiking as the comment on
> the graph says. 500 regions ain't too many...
>
> How many servers in your cluster?
>
> St.Ack
>
>
>>
>>
>
Re: Issue with StoreFiles with bulk import.
Posted by Stack <st...@duboce.net>.
On Fri, Aug 13, 2010 at 6:26 AM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> The main issue is that the stores get very fragmented. I've seen as much as 300 storeFiles for one region during this process.
How many column families and doesn't this number start to do down
after a while because they get compacted together? How many regions
per server do you have?
I'm concerned about performance with that many files to search
through. We are seeing this in the UI for a regions list
(storeFiles=?), also we see this through our JMX monitoring as well.
If there is no issue with performance I'm OK. But as it seems right
now the only way I can decrease the storeFile count is do to a major
compaction.
You are doing manual intervention when you do that... hbase should be
compacting in the background bring down the overall numbers of
storefiles per region.
We only do one major compaction per day to minimize load on the
system. So during the day am I going to see drastically reduced
performance with 30,000 storeFiles versus 800 when it's major
compacted? The 90 second timeout was the flush wait timeout. We are
inserting pretty fast so the flush timeout hits it's 90 second limit
and aborts.
Have you changed other configurations that disable the natural hbase
compacting facility?
St.Ack
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
> Sent: Friday, August 13, 2010 12:33 AM
> To: user@hbase.apache.org
> Subject: Re: Issue with StoreFiles with bulk import.
>
> On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
> <je...@networkedinsights.com> wrote:
>> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
>
> Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
>
>
>> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
>
>
> So when you mapreduce using these big files as source and insert into
> hbase, its not an issue?
>
>
>> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
>
>
> Why you need to change?
>
>
>> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
>
> Probably. Have you tinkered with hbase default settings in any way?
>
> Perhaps you are getting better parallelism when lots of small files to
> chomp on? More concurrent maps/clients? So rate of upload goes up?
>
>
>> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>>
>
>
> The 90 second timeout is the regionserver timing out against
> zookeeper? Or is it something else?
>
> Storefiles are on the filesystem so what do you mean by the above fear
> of their not being committed to disk?
>
>
>> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>>
>> http://twitpic.com/photos/jeremy_carroll
>>
>
>
> Nice pictures.
>
> 30k storefiles is a good number. They will go up as you are doing a
> bulk load as the compactor is probably overrun. HBase will usually
> catch up though especially after the upload completes.
>
> Do you have compression enabled?
>
> I see regions growing steadily rather than spiking as the comment on
> the graph says. 500 regions ain't too many...
>
> How many servers in your cluster?
>
> St.Ack
>
>
>>
>>
>
Re: Issue with StoreFiles with bulk import.
Posted by Stack <st...@duboce.net>.
Table flushCommit flushes the client-side buffering of writes. That
is different from server flush of its memstore when they hit size
limits, the log message you are seeing below. The two are not
directly related.
On Fri, Aug 13, 2010 at 6:47 AM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> I'm wondering if our code is calling tableCommit too frequently and flushing the regions to disk when it does not have too. I'm seeing a lot of these entries.
>
> 2010-08-13 08:44:44,228 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_FLUSH: table,20100812 d92a4edf020575f5f7209e54981cc6b97b82d2b8,1281662636511.eea5d3079cb2d7c8637c725947e87428.
>
> Also for the job counters I see that in the map phase we called NUMBER_OF_TABLE_COMMITS 3,049 times. Does calling tableCommit flush the region to disk and create a new store file? If so, I believe we can just move the tableCommit to the reduce phase instead of during the map phase.
>
I'd say uses smallish buffer -- 2M -- and then just let hbase manage
flushing. When map is done flushcommit should be called to clean out
anything left in client-side flush buffers.
St.Ack
>
> ________________________________________
> From: Jeremy Carroll [jeremy.carroll@networkedinsights.com]
> Sent: Friday, August 13, 2010 8:26 AM
> To: user@hbase.apache.org
> Subject: RE: Issue with StoreFiles with bulk import.
>
> The main issue is that the stores get very fragmented. I've seen as much as 300 storeFiles for one region during this process. I'm concerned about performance with that many files to search through. We are seeing this in the UI for a regions list (storeFiles=?), also we see this through our JMX monitoring as well. If there is no issue with performance I'm OK. But as it seems right now the only way I can decrease the storeFile count is do to a major compaction. We only do one major compaction per day to minimize load on the system. So during the day am I going to see drastically reduced performance with 30,000 storeFiles versus 800 when it's major compacted? The 90 second timeout was the flush wait timeout. We are inserting pretty fast so the flush timeout hits it's 90 second limit and aborts.
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
> Sent: Friday, August 13, 2010 12:33 AM
> To: user@hbase.apache.org
> Subject: Re: Issue with StoreFiles with bulk import.
>
> On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
> <je...@networkedinsights.com> wrote:
>> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
>
> Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
>
>
>> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
>
>
> So when you mapreduce using these big files as source and insert into
> hbase, its not an issue?
>
>
>> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
>
>
> Why you need to change?
>
>
>> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
>
> Probably. Have you tinkered with hbase default settings in any way?
>
> Perhaps you are getting better parallelism when lots of small files to
> chomp on? More concurrent maps/clients? So rate of upload goes up?
>
>
>> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>>
>
>
> The 90 second timeout is the regionserver timing out against
> zookeeper? Or is it something else?
>
> Storefiles are on the filesystem so what do you mean by the above fear
> of their not being committed to disk?
>
>
>> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>>
>> http://twitpic.com/photos/jeremy_carroll
>>
>
>
> Nice pictures.
>
> 30k storefiles is a good number. They will go up as you are doing a
> bulk load as the compactor is probably overrun. HBase will usually
> catch up though especially after the upload completes.
>
> Do you have compression enabled?
>
> I see regions growing steadily rather than spiking as the comment on
> the graph says. 500 regions ain't too many...
>
> How many servers in your cluster?
>
> St.Ack
>
>
>>
>>
>
RE: Issue with StoreFiles with bulk import.
Posted by Jeremy Carroll <je...@networkedinsights.com>.
I'm wondering if our code is calling tableCommit too frequently and flushing the regions to disk when it does not have too. I'm seeing a lot of these entries.
2010-08-13 08:44:44,228 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_FLUSH: table,20100812 d92a4edf020575f5f7209e54981cc6b97b82d2b8,1281662636511.eea5d3079cb2d7c8637c725947e87428.
Also for the job counters I see that in the map phase we called NUMBER_OF_TABLE_COMMITS 3,049 times. Does calling tableCommit flush the region to disk and create a new store file? If so, I believe we can just move the tableCommit to the reduce phase instead of during the map phase.
________________________________________
From: Jeremy Carroll [jeremy.carroll@networkedinsights.com]
Sent: Friday, August 13, 2010 8:26 AM
To: user@hbase.apache.org
Subject: RE: Issue with StoreFiles with bulk import.
The main issue is that the stores get very fragmented. I've seen as much as 300 storeFiles for one region during this process. I'm concerned about performance with that many files to search through. We are seeing this in the UI for a regions list (storeFiles=?), also we see this through our JMX monitoring as well. If there is no issue with performance I'm OK. But as it seems right now the only way I can decrease the storeFile count is do to a major compaction. We only do one major compaction per day to minimize load on the system. So during the day am I going to see drastically reduced performance with 30,000 storeFiles versus 800 when it's major compacted? The 90 second timeout was the flush wait timeout. We are inserting pretty fast so the flush timeout hits it's 90 second limit and aborts.
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Friday, August 13, 2010 12:33 AM
To: user@hbase.apache.org
Subject: Re: Issue with StoreFiles with bulk import.
On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
So when you mapreduce using these big files as source and insert into
hbase, its not an issue?
> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
Why you need to change?
> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
Probably. Have you tinkered with hbase default settings in any way?
Perhaps you are getting better parallelism when lots of small files to
chomp on? More concurrent maps/clients? So rate of upload goes up?
> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>
The 90 second timeout is the regionserver timing out against
zookeeper? Or is it something else?
Storefiles are on the filesystem so what do you mean by the above fear
of their not being committed to disk?
> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>
> http://twitpic.com/photos/jeremy_carroll
>
Nice pictures.
30k storefiles is a good number. They will go up as you are doing a
bulk load as the compactor is probably overrun. HBase will usually
catch up though especially after the upload completes.
Do you have compression enabled?
I see regions growing steadily rather than spiking as the comment on
the graph says. 500 regions ain't too many...
How many servers in your cluster?
St.Ack
>
>
RE: Issue with StoreFiles with bulk import.
Posted by Jeremy Carroll <je...@networkedinsights.com>.
The main issue is that the stores get very fragmented. I've seen as much as 300 storeFiles for one region during this process. I'm concerned about performance with that many files to search through. We are seeing this in the UI for a regions list (storeFiles=?), also we see this through our JMX monitoring as well. If there is no issue with performance I'm OK. But as it seems right now the only way I can decrease the storeFile count is do to a major compaction. We only do one major compaction per day to minimize load on the system. So during the day am I going to see drastically reduced performance with 30,000 storeFiles versus 800 when it's major compacted? The 90 second timeout was the flush wait timeout. We are inserting pretty fast so the flush timeout hits it's 90 second limit and aborts.
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Friday, August 13, 2010 12:33 AM
To: user@hbase.apache.org
Subject: Re: Issue with StoreFiles with bulk import.
On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
So when you mapreduce using these big files as source and insert into
hbase, its not an issue?
> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
Why you need to change?
> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
Probably. Have you tinkered with hbase default settings in any way?
Perhaps you are getting better parallelism when lots of small files to
chomp on? More concurrent maps/clients? So rate of upload goes up?
> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>
The 90 second timeout is the regionserver timing out against
zookeeper? Or is it something else?
Storefiles are on the filesystem so what do you mean by the above fear
of their not being committed to disk?
> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>
> http://twitpic.com/photos/jeremy_carroll
>
Nice pictures.
30k storefiles is a good number. They will go up as you are doing a
bulk load as the compactor is probably overrun. HBase will usually
catch up though especially after the upload completes.
Do you have compression enabled?
I see regions growing steadily rather than spiking as the comment on
the graph says. 500 regions ain't too many...
How many servers in your cluster?
St.Ack
>
>
Re: Issue with StoreFiles with bulk import.
Posted by Stack <st...@duboce.net>.
On Thu, Aug 12, 2010 at 1:13 PM, Jeremy Carroll
<je...@networkedinsights.com> wrote:
> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created.
Where you see this Jeremy? In the UI? What kinda numbers are you seeing?
> We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand.
So when you mapreduce using these big files as source and insert into
hbase, its not an issue?
> When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically.
Why you need to change?
> I do not know if this is happening because we are flushing the commits more frequently with smaller files.
Probably. Have you tinkered with hbase default settings in any way?
Perhaps you are getting better parallelism when lots of small files to
chomp on? More concurrent maps/clients? So rate of upload goes up?
> I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>
The 90 second timeout is the regionserver timing out against
zookeeper? Or is it something else?
Storefiles are on the filesystem so what do you mean by the above fear
of their not being committed to disk?
> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>
> http://twitpic.com/photos/jeremy_carroll
>
Nice pictures.
30k storefiles is a good number. They will go up as you are doing a
bulk load as the compactor is probably overrun. HBase will usually
catch up though especially after the upload completes.
Do you have compression enabled?
I see regions growing steadily rather than spiking as the comment on
the graph says. 500 regions ain't too many...
How many servers in your cluster?
St.Ack
>
>
Re: Issue with StoreFiles with bulk import.
Posted by Lekhnath <lb...@veriskhealth.com>.
On 8/13/2010 1:58 AM, Jeremy Carroll wrote:
> I'm currently importing some files into HBase and am running into an problem with a large number of store files being created. We have some back data which is stored in very large sequence files (3-5 Gb in size). When we import this data the amount of stores created does not get out of hand. When we switch to smaller sequence files being imported we see that the number of stores rises quite dramatically. I do not know if this is happening because we are flushing the commits more frequently with smaller files. I'm wondering if anybody has any advice regarding this issue. My main concern is during this process we do not finish flushing to disk (And we set WritetoWal False). We always hit the 90 second timeout due to heavy write load. As these store files pile up, and they do not get committed to disk, we run into issues where we could lose a lot of data if something were to crash.
>
> I have created screen shots of or monitoring application for HBase which shows the spikes in activity.
>
> http://twitpic.com/photos/jeremy_carroll
>
>
>
We faced the similar problem while doing bulk imports. For large number
of reducers, we got large number of small files. Most probably, each
reducer creates one file at the list. Making appropriate number of
reducers and input file size solved the issue.
Lekhnath
This email is intended for the recipient only. If you are not the intended
recipient please disregard, and do not use the information for any purpose.