You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 谢良 <xi...@xiaomi.com> on 2013/06/27 10:21:12 UTC
答复: flushing + compactions after config change
If reached memstore global up-limit, you'll find "Blocking updates on" in your files(see MemStoreFlusher.reclaimMemStoreMemory);
If it's caused by too many log files, you'll find "Too many hlogs: logs="(see HLog.cleanOldLogs)
Hope it's helpful for you:)
Best,
Liang
________________________________________
发件人: Viral Bajaria [viral.bajaria@gmail.com]
发送时间: 2013年6月27日 16:18
收件人: user@hbase.apache.org
主题: Re: flushing + compactions after config change
Thanks for the quick response Anoop.
The current memstore reserved (IIRC) would be 0.35 of total heap right ?
The RS total heap is 10231MB, used is at 5000MB. Total number of regions is
217 and there are approx 150 regions with 2 families, ~60 with 1 family and
remaining with 3 families.
How to check if the flushes are due to too many WAL files ? Does it get
logged ?
Thanks,
Viral
On Thu, Jun 27, 2013 at 12:51 AM, Anoop John <an...@gmail.com> wrote:
> You mean there is enough memstore reserved heap in the RS, so that there
> wont be premature flushes because of global heap pressure? What is the RS
> max mem and how many regions and CFs in each? Can you check whether the
> flushes happening because of too many WAL files?
>
> -Anoop-
>
答复: 答复: flushing + compactions after config change
Posted by 谢良 <xi...@xiaomi.com>.
I have no more good ideas on it, maybe your should throttle the client write request ?,i guess the cluster is on a heavy write request status. when you saw "Too many hlogs" continually, then probably you'll see the numOfStoreFiles is high also, due to the forcing flush region(beginning from the oldest one), then compaction queue size will increase as well...
Best,
Liang
________________________________________
发件人: Viral Bajaria [viral.bajaria@gmail.com]
发送时间: 2013年6月27日 16:29
收件人: user@hbase.apache.org
主题: Re: 答复: flushing + compactions after config change
Thanks Liang!
Found the logs. I had gone overboard with my grep's and missed the "Too
many hlogs" line for the regions that I was trying to debug.
A few sample log lines:
2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14
2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
0e940167482d42f1999b29a023c7c18a
2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08
2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s):
0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14,
e380fd8a7174d34feb903baa97564e08
Any pointers on what's the best practice for avoiding this scenario ?
Thanks,
Viral
On Thu, Jun 27, 2013 at 1:21 AM, 谢良 <xi...@xiaomi.com> wrote:
> If reached memstore global up-limit, you'll find "Blocking updates on"
> in your files(see MemStoreFlusher.reclaimMemStoreMemory);
> If it's caused by too many log files, you'll find "Too many hlogs:
> logs="(see HLog.cleanOldLogs)
> Hope it's helpful for you:)
>
> Best,
> Liang
>
Re: 答复: flushing + compactions after config change
Posted by Anoop John <an...@gmail.com>.
Viral,
Basically when you increase the memstore flush size ( well ur aim
there is to reduce flushes and make data sit in memory for longer time) you
need to carefully consider the 2 things
1. What is the max heap and what is the % memory you have allocated max for
all the memstores in a RS. And see how many regions and CFs and whether the
allocated max memstore memory is enough. Basically you should not make
premature flushes because of global heap preassure. As you said this will
create many small HFiles and so compactions
2. Check on the max Hlogs setting. Becuase of too many hlogs also there can
be a forced flush which also can create small HFiles. The default value is
32. The HLOg rolling will happen when one HLog's size reaches 95% size of
ur HDFS block size. (This 95% is also configurable) If you have increased
the flush size and heap allocated for memstores, still you can get
premature flushes because of this HLog stuff.
Also from 94 onwards there is compression feature for HLog.. Have a look
at that also.
-Anoop-
On Fri, Jun 28, 2013 at 6:52 AM, Viral Bajaria <vi...@gmail.com>wrote:
> Thanks for the detailed JVM OPTS. On a side note, I doubt the issue with
> too many hlogs has anything to do with that right ? Because I have enough
> spare heap space and my Memstore's are not at the lowerLimit of the total
> available heap (i.e. 0.35)
>
> -Viral
>
> On Thu, Jun 27, 2013 at 6:09 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> > Hope that helpful.
>
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
Thanks for the detailed JVM OPTS. On a side note, I doubt the issue with
too many hlogs has anything to do with that right ? Because I have enough
spare heap space and my Memstore's are not at the lowerLimit of the total
available heap (i.e. 0.35)
-Viral
On Thu, Jun 27, 2013 at 6:09 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hope that helpful.
Re: 答复: flushing + compactions after config change
Posted by Azuryy Yu <az...@gmail.com>.
Hi Viral,
the following are all needed for CMS:
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:CMSInitiatingOccupancyFraction=70
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled
-XX:CMSMaxAbortablePrecleanTime=300
-XX:+CMSScavengeBeforeRemark
and if your JDK version is greater than 1.6.23, then add :
-XX:+UseCompressedOops
-XX:SoftRefLRUPolicyMSPerMB=0
and you'd better add GC log
-verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-Xloggc:$HBASE_LOG_DIR/gc.log
if your JDK version is greater than 1.6.23, then add :
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=9
-XX:GCLogFileSize=20m
Hope that helpful.
On Fri, Jun 28, 2013 at 5:06 AM, Viral Bajaria <vi...@gmail.com>wrote:
> Thanks Azuryy. Look forward to it.
>
> Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created
> ? Tried looking around but could not find the details.
>
> On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>
> > your JVM options arenot enough. I will give you some detail when I go
> back
> > office tomorrow.
> >
> > --Send from my Sony mobile.
> >
>
Re: 答复: flushing + compactions after config change
Posted by Himanshu Vashishtha <hv...@gmail.com>.
bq.
"> On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <vi...@gmail.com>
> wrote:
> It's not random, it picks the region with the most data in its memstores.
>
That's weird, because I see some of my regions which receive the least
amount of data in a given time period flushing before the regions that are
receiving data continuously."
I agree with Viral here. When max logs are reached, we look at the oldest
wal and see which regions should be flushed in order to get that first wal
(read oldest) archived. In your case Viral, these regions could be those
which are not receiving many edits when 32 logs have been rolled.
It may be very specific to your use case, but you could try playing with
max number of logs? May be make them 16, 40, etc?
On Fri, Jun 28, 2013 at 4:53 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> On Fri, Jun 28, 2013 at 2:39 PM, Viral Bajaria <vi...@gmail.com>
> wrote:
> > On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <viral.bajaria@gmail.com
> >
> >> wrote:
> >> It's not random, it picks the region with the most data in its
> memstores.
> >>
> >
> > That's weird, because I see some of my regions which receive the least
> > amount of data in a given time period flushing before the regions that
> are
> > receiving data continuously. The reason I know this is because of the
> write
> > pattern. Some of my tables are in catch-up mode i.e. I am ingesting data
> > from the past and they always have something to do. While some tables are
> > not in catch-up mode and are just sitting idle for most of the time. Yet
> I
> > see high number of flushes for those regions too.
> >
> >
> >>
> >> I doubt that it's the fact that it's a major compaction that it's
> >> making everything worse. When a minor gets promoted into a major it's
> >> because we're already going to compact all the files, so we might as
> >> well get rid of some deletes at the same time. They are all getting
> >> selected because the files are within the selection ratio. I would not
> >> focus on this to resolve your problem.
> >>
> >
> > I meant worse for my writes not for HBase as a whole.
> >
> >
> >>
> >> I haven't been closely following this thread, but have you posted a
> >> log snippet somewhere? It's usually much more telling and we eliminate
> >> a few levels of interpretation. Make sure it's at DEBUG, and that you
> >> grab a few hours of activity. Get the GC log for the same time as
> >> well. Drop this on a web server or pastebin if it fits.
> >>
> >
> > The only log snippet that I posted was the flushing action. Also that log
> > was not everything, I had grep'd a few lines out. Let me collect some
> more
> > stats here and post it again. I just enabled GC logging on this server,
> > deployed the wrong config out initially which had no GC logging. I am not
> > sure how GC logs will help here given that I am at less than 50% heap
> space
> > used and so I would doubt a stop the world GC happening. Are you trying
> to
> > look for some other information ?
>
> Just trying to cover all the bases.
>
> J-D
>
Re: 答复: flushing + compactions after config change
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Fri, Jun 28, 2013 at 2:39 PM, Viral Bajaria <vi...@gmail.com> wrote:
> On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <vi...@gmail.com>
>> wrote:
>> It's not random, it picks the region with the most data in its memstores.
>>
>
> That's weird, because I see some of my regions which receive the least
> amount of data in a given time period flushing before the regions that are
> receiving data continuously. The reason I know this is because of the write
> pattern. Some of my tables are in catch-up mode i.e. I am ingesting data
> from the past and they always have something to do. While some tables are
> not in catch-up mode and are just sitting idle for most of the time. Yet I
> see high number of flushes for those regions too.
>
>
>>
>> I doubt that it's the fact that it's a major compaction that it's
>> making everything worse. When a minor gets promoted into a major it's
>> because we're already going to compact all the files, so we might as
>> well get rid of some deletes at the same time. They are all getting
>> selected because the files are within the selection ratio. I would not
>> focus on this to resolve your problem.
>>
>
> I meant worse for my writes not for HBase as a whole.
>
>
>>
>> I haven't been closely following this thread, but have you posted a
>> log snippet somewhere? It's usually much more telling and we eliminate
>> a few levels of interpretation. Make sure it's at DEBUG, and that you
>> grab a few hours of activity. Get the GC log for the same time as
>> well. Drop this on a web server or pastebin if it fits.
>>
>
> The only log snippet that I posted was the flushing action. Also that log
> was not everything, I had grep'd a few lines out. Let me collect some more
> stats here and post it again. I just enabled GC logging on this server,
> deployed the wrong config out initially which had no GC logging. I am not
> sure how GC logs will help here given that I am at less than 50% heap space
> used and so I would doubt a stop the world GC happening. Are you trying to
> look for some other information ?
Just trying to cover all the bases.
J-D
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <vi...@gmail.com>
> wrote:
> It's not random, it picks the region with the most data in its memstores.
>
That's weird, because I see some of my regions which receive the least
amount of data in a given time period flushing before the regions that are
receiving data continuously. The reason I know this is because of the write
pattern. Some of my tables are in catch-up mode i.e. I am ingesting data
from the past and they always have something to do. While some tables are
not in catch-up mode and are just sitting idle for most of the time. Yet I
see high number of flushes for those regions too.
>
> I doubt that it's the fact that it's a major compaction that it's
> making everything worse. When a minor gets promoted into a major it's
> because we're already going to compact all the files, so we might as
> well get rid of some deletes at the same time. They are all getting
> selected because the files are within the selection ratio. I would not
> focus on this to resolve your problem.
>
I meant worse for my writes not for HBase as a whole.
>
> I haven't been closely following this thread, but have you posted a
> log snippet somewhere? It's usually much more telling and we eliminate
> a few levels of interpretation. Make sure it's at DEBUG, and that you
> grab a few hours of activity. Get the GC log for the same time as
> well. Drop this on a web server or pastebin if it fits.
>
The only log snippet that I posted was the flushing action. Also that log
was not everything, I had grep'd a few lines out. Let me collect some more
stats here and post it again. I just enabled GC logging on this server,
deployed the wrong config out initially which had no GC logging. I am not
sure how GC logs will help here given that I am at less than 50% heap space
used and so I would doubt a stop the world GC happening. Are you trying to
look for some other information ?
> Thx,
>
> J-D
Thanks to everyone on this list for their time!
-Viral
Re: 答复: flushing + compactions after config change
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <vi...@gmail.com> wrote:
> Hey JD,
>
> Thanks for the clarification. I also came across a previous thread which
> sort of talks about a similar problem.
> http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3CCAGpTDNfWNRsNqV7n3wgjE-iCHZPx-CXn1TBchgwRPOhgcoS+bw@mail.gmail.com%3E
>
> I guess my problem is also similar to the fact that my writes are well
> distributed and at a given time I could be writing to a lot of regions.
> Some of the regions receive very little data but since the flush algorithm
> choose at random what to flush when "too many hlogs" is hit, it will flush
It's not random, it picks the region with the most data in its memstores.
> a region with less than 10mb of data causing too many small files. This
> in-turn causes compaction storms where even though major compactions is
> disabled, some of the minor get upgraded to major and that's when things
> start getting worse.
I doubt that it's the fact that it's a major compaction that it's
making everything worse. When a minor gets promoted into a major it's
because we're already going to compact all the files, so we might as
well get rid of some deletes at the same time. They are all getting
selected because the files are within the selection ratio. I would not
focus on this to resolve your problem.
>
> My compaction queues are still the same and so I doubt I will be coming out
> of this storm without bumping up max hlogs for now. Reducing regions per
> server is one option but then I will be wasting my resources since the
> servers at current load are at < 30% CPU and < 25% RAM. Maybe I can bump up
> heap space and give more memory to the the memstore. Sorry, I am just
> thinking out loud.
I haven't been closely following this thread, but have you posted a
log snippet somewhere? It's usually much more telling and we eliminate
a few levels of interpretation. Make sure it's at DEBUG, and that you
grab a few hours of activity. Get the GC log for the same time as
well. Drop this on a web server or pastebin if it fits.
Thx,
J-D
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
Hey JD,
Thanks for the clarification. I also came across a previous thread which
sort of talks about a similar problem.
http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3CCAGpTDNfWNRsNqV7n3wgjE-iCHZPx-CXn1TBchgwRPOhgcoS+bw@mail.gmail.com%3E
I guess my problem is also similar to the fact that my writes are well
distributed and at a given time I could be writing to a lot of regions.
Some of the regions receive very little data but since the flush algorithm
choose at random what to flush when "too many hlogs" is hit, it will flush
a region with less than 10mb of data causing too many small files. This
in-turn causes compaction storms where even though major compactions is
disabled, some of the minor get upgraded to major and that's when things
start getting worse.
My compaction queues are still the same and so I doubt I will be coming out
of this storm without bumping up max hlogs for now. Reducing regions per
server is one option but then I will be wasting my resources since the
servers at current load are at < 30% CPU and < 25% RAM. Maybe I can bump up
heap space and give more memory to the the memstore. Sorry, I am just
thinking out loud.
Thanks,
Viral
On Thu, Jun 27, 2013 at 2:40 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> No, all your data eventually makes it into the log, just potentially
> not as quickly :)
>
Re: 答复: flushing + compactions after config change
Posted by Jean-Daniel Cryans <jd...@apache.org>.
No, all your data eventually makes it into the log, just potentially
not as quickly :)
J-D
On Thu, Jun 27, 2013 at 2:06 PM, Viral Bajaria <vi...@gmail.com> wrote:
> Thanks Azuryy. Look forward to it.
>
> Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created
> ? Tried looking around but could not find the details.
>
> On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> your JVM options arenot enough. I will give you some detail when I go back
>> office tomorrow.
>>
>> --Send from my Sony mobile.
>>
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
Thanks Azuryy. Look forward to it.
Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created
? Tried looking around but could not find the details.
On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> your JVM options arenot enough. I will give you some detail when I go back
> office tomorrow.
>
> --Send from my Sony mobile.
>
Re: 答复: flushing + compactions after config change
Posted by Azuryy Yu <az...@gmail.com>.
your JVM options arenot enough. I will give you some detail when I go back
office tomorrow.
--Send from my Sony mobile.
On Jun 27, 2013 6:09 PM, "Viral Bajaria" <vi...@gmail.com> wrote:
> I do have a heavy write operation going on. Actually heavy is relative. Not
> all tables/regions are seeing the same amount of writes at the same time.
> There is definitely a burst of writes that can happen on some regions. In
> addition to that there are some processing jobs which play catch up and
> could be processing data in the past and they could have more heavy write
> operations.
>
> I think my main problem is, my writes are well distributed across regions.
> A batch of puts most probably end up hitting every region since they get
> distributed fairly well. In that scenario, I am guessing I get a lot of
> WALs though I am just speculating.
>
> Regarding the JVM options (minus some settings for remote profiling):
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M
>
> On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu <az...@gmail.com> wrote:
>
> > Can you paste your JVM options here? and Do you have an extensive write
> on
> > your hbase cluster?
> >
>
答复: 答复: flushing + compactions after config change
Posted by 谢良 <xi...@xiaomi.com>.
btw, don't use "CMSIncrementalMode", iirc, it had been removed from hotspot upstream accually.
________________________________________
发件人: Viral Bajaria [viral.bajaria@gmail.com]
发送时间: 2013年6月27日 18:08
收件人: user@hbase.apache.org
主题: Re: 答复: flushing + compactions after config change
I do have a heavy write operation going on. Actually heavy is relative. Not
all tables/regions are seeing the same amount of writes at the same time.
There is definitely a burst of writes that can happen on some regions. In
addition to that there are some processing jobs which play catch up and
could be processing data in the past and they could have more heavy write
operations.
I think my main problem is, my writes are well distributed across regions.
A batch of puts most probably end up hitting every region since they get
distributed fairly well. In that scenario, I am guessing I get a lot of
WALs though I am just speculating.
Regarding the JVM options (minus some settings for remote profiling):
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M
On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu <az...@gmail.com> wrote:
> Can you paste your JVM options here? and Do you have an extensive write on
> your hbase cluster?
>
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
I do have a heavy write operation going on. Actually heavy is relative. Not
all tables/regions are seeing the same amount of writes at the same time.
There is definitely a burst of writes that can happen on some regions. In
addition to that there are some processing jobs which play catch up and
could be processing data in the past and they could have more heavy write
operations.
I think my main problem is, my writes are well distributed across regions.
A batch of puts most probably end up hitting every region since they get
distributed fairly well. In that scenario, I am guessing I get a lot of
WALs though I am just speculating.
Regarding the JVM options (minus some settings for remote profiling):
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M
On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu <az...@gmail.com> wrote:
> Can you paste your JVM options here? and Do you have an extensive write on
> your hbase cluster?
>
Re: 答复: flushing + compactions after config change
Posted by Azuryy Yu <az...@gmail.com>.
Can you paste your JVM options here? and Do you have an extensive write on
your hbase cluster?
On Thu, Jun 27, 2013 at 5:47 PM, Viral Bajaria <vi...@gmail.com>wrote:
> 0.94.4 with plans to upgrade to the latest 0.94 release.
>
> On Thu, Jun 27, 2013 at 2:22 AM, Azuryy Yu <az...@gmail.com> wrote:
>
> > hey Viral,
> > Which hbase version are you using?
> >
>
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
0.94.4 with plans to upgrade to the latest 0.94 release.
On Thu, Jun 27, 2013 at 2:22 AM, Azuryy Yu <az...@gmail.com> wrote:
> hey Viral,
> Which hbase version are you using?
>
Re: 答复: flushing + compactions after config change
Posted by Azuryy Yu <az...@gmail.com>.
hey Viral,
Which hbase version are you using?
On Thu, Jun 27, 2013 at 5:03 PM, Anoop John <an...@gmail.com> wrote:
> The config "hbase.regionserver.maxlogs" specifies what is the max #logs and
> defaults to 32. But remember if there are so many log files to replay then
> the MTTR will become more (RS down case )
>
> -Anoop-
> On Thu, Jun 27, 2013 at 1:59 PM, Viral Bajaria <viral.bajaria@gmail.com
> >wrote:
>
> > Thanks Liang!
> >
> > Found the logs. I had gone overboard with my grep's and missed the "Too
> > many hlogs" line for the regions that I was trying to debug.
> >
> > A few sample log lines:
> >
> > 2013-06-27 07:42:49,602 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> > Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
> > 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14
> > 2013-06-27 08:10:29,996 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> > Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
> > 0e940167482d42f1999b29a023c7c18a
> > 2013-06-27 08:17:44,719 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> > Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
> > 0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08
> > 2013-06-27 08:23:45,357 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> > Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s):
> > 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14,
> > e380fd8a7174d34feb903baa97564e08
> >
> > Any pointers on what's the best practice for avoiding this scenario ?
> >
> > Thanks,
> > Viral
> >
> > On Thu, Jun 27, 2013 at 1:21 AM, 谢良 <xi...@xiaomi.com> wrote:
> >
> > > If reached memstore global up-limit, you'll find "Blocking updates
> on"
> > > in your files(see MemStoreFlusher.reclaimMemStoreMemory);
> > > If it's caused by too many log files, you'll find "Too many hlogs:
> > > logs="(see HLog.cleanOldLogs)
> > > Hope it's helpful for you:)
> > >
> > > Best,
> > > Liang
> > >
> >
>
Re: 答复: flushing + compactions after config change
Posted by Anoop John <an...@gmail.com>.
The config "hbase.regionserver.maxlogs" specifies what is the max #logs and
defaults to 32. But remember if there are so many log files to replay then
the MTTR will become more (RS down case )
-Anoop-
On Thu, Jun 27, 2013 at 1:59 PM, Viral Bajaria <vi...@gmail.com>wrote:
> Thanks Liang!
>
> Found the logs. I had gone overboard with my grep's and missed the "Too
> many hlogs" line for the regions that I was trying to debug.
>
> A few sample log lines:
>
> 2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
> 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14
> 2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
> 0e940167482d42f1999b29a023c7c18a
> 2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
> 0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08
> 2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s):
> 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14,
> e380fd8a7174d34feb903baa97564e08
>
> Any pointers on what's the best practice for avoiding this scenario ?
>
> Thanks,
> Viral
>
> On Thu, Jun 27, 2013 at 1:21 AM, 谢良 <xi...@xiaomi.com> wrote:
>
> > If reached memstore global up-limit, you'll find "Blocking updates on"
> > in your files(see MemStoreFlusher.reclaimMemStoreMemory);
> > If it's caused by too many log files, you'll find "Too many hlogs:
> > logs="(see HLog.cleanOldLogs)
> > Hope it's helpful for you:)
> >
> > Best,
> > Liang
> >
>
Re: 答复: flushing + compactions after config change
Posted by Viral Bajaria <vi...@gmail.com>.
Thanks Liang!
Found the logs. I had gone overboard with my grep's and missed the "Too
many hlogs" line for the regions that I was trying to debug.
A few sample log lines:
2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14
2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
0e940167482d42f1999b29a023c7c18a
2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s):
0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08
2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s):
0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14,
e380fd8a7174d34feb903baa97564e08
Any pointers on what's the best practice for avoiding this scenario ?
Thanks,
Viral
On Thu, Jun 27, 2013 at 1:21 AM, 谢良 <xi...@xiaomi.com> wrote:
> If reached memstore global up-limit, you'll find "Blocking updates on"
> in your files(see MemStoreFlusher.reclaimMemStoreMemory);
> If it's caused by too many log files, you'll find "Too many hlogs:
> logs="(see HLog.cleanOldLogs)
> Hope it's helpful for you:)
>
> Best,
> Liang
>