You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/06/23 04:00:16 UTC
Hmaster crashes caused by splitting log.
Because Master usually<javascript:void(0)> uses little memory. So its memory is 4G.
DFS block is 256M and hbase.regionserver.maxlogs is 32. One region server can save max 8G Hlog.
In my performance cluster(0.90.3), The Hmaster memory from 100 M up to 4G when one region server crashed.
I dug it and found the flow control does not work when write thread is normal.
// If we crossed the chunk threshold, wait for more space to be available
synchronized (dataAvailable) {
while (totalBuffered > maxHeapUsage && thrown == null) {
LOG.debug("Used " + totalBuffered + " bytes of buffered edits, waiting for IO threads...");
dataAvailable.wait(3000);
}
dataAvailable.notifyAll();
}
If the code is below. It seems better.
// If we crossed the chunk threshold, wait for more space to be available
synchronized (dataAvailable) {
while (totalBuffered > maxHeapUsage) {
LOG.debug("Used " + totalBuffered + " bytes of buffered edits, waiting for IO threads...");
dataAvailable.wait(3000);
}
dataAvailable.notifyAll();
}
Re: Hmaster crashes caused by splitting log.
Posted by Gaojinchao <ga...@huawei.com>.
Thanks Ted.
I didn't see the log . I have filed a issue(HBASE-4028)) and uploaded the log.
I will make a patch and please review it.
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年6月24日 13:24
收件人: dev@hbase.apache.org
主题: Re: Hmaster crashes caused by splitting log.
In doRun():
try {
writeBuffer(buffer);
} finally {
entryBuffers.doneWriting(buffer);
}
totalBuffered being negative implies that there were exceptions in
writeBuffer().
Did you see the following in log ?
e = RemoteExceptionHandler.checkIOException(e);
LOG.fatal(this.getName() + " Got while writing log entry to log",
e);
Thanks
On Thu, Jun 23, 2011 at 10:10 PM, Gaojinchao <ga...@huawei.com> wrote:
> I am some doubt. It seems thrown object isn't null ,But , why the top can't
> catch?
> My guess is Thread re-entrant and add some logs. It show that value of
> totalBuffered is -565832
>
>
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -540664
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -540664release size68872
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -565832
>
>
>
>
Re: Hmaster crashes caused by splitting log.
Posted by Ted Yu <yu...@gmail.com>.
In doRun():
try {
writeBuffer(buffer);
} finally {
entryBuffers.doneWriting(buffer);
}
totalBuffered being negative implies that there were exceptions in
writeBuffer().
Did you see the following in log ?
e = RemoteExceptionHandler.checkIOException(e);
LOG.fatal(this.getName() + " Got while writing log entry to log",
e);
Thanks
On Thu, Jun 23, 2011 at 10:10 PM, Gaojinchao <ga...@huawei.com> wrote:
> I am some doubt. It seems thrown object isn't null ,But , why the top can't
> catch?
> My guess is Thread re-entrant and add some logs. It show that value of
> totalBuffered is -565832
>
>
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -540664
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -540664release size68872
> hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used
> -565832
>
>
>
>
Re: Hmaster crashes caused by splitting log.
Posted by Gaojinchao <ga...@huawei.com>.
I am some doubt. It seems thrown object isn't null ,But , why the top can't catch?
My guess is Thread re-entrant and add some logs. It show that value of totalBuffered is -565832
hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used -540664
hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used -540664release size68872
hbase-root-master-157-5-111-21.log:2011-06-24 10:29:52,119 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: gjc:release Used -565832
Re: Hmaster crashes caused by splitting log.
Posted by Ted Yu <yu...@gmail.com>.
I think you can place this condition for dataAvailable.notifyAll():
if (thrown == null) {
dataAvailable.notifyAll();
}
2011/6/22 Gaojinchao <ga...@huawei.com>
> Thanks, You are right. :)
>
> I will test it.
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2011年6月23日 11:00
> 收件人: dev@hbase.apache.org
> 主题: Re: Hmaster crashes caused by splitting log.
>
> I guess you have seen this call directly below the following code snippet:
> checkForErrors();
> I think checking thrown should be kept in the condition.
>
> I assume we can lift the above call before dataAvailable.notifyAll()
>
> My 2 cents.
>
> On Wed, Jun 22, 2011 at 7:00 PM, Gaojinchao <ga...@huawei.com> wrote:
>
> >
> > Because Master usually<javascript:void(0)> uses little memory. So its
> > memory is 4G.
> >
> > DFS block is 256M and hbase.regionserver.maxlogs is 32. One region server
> > can save max 8G Hlog.
> >
> > In my performance cluster(0.90.3), The Hmaster memory from 100 M up to 4G
> > when one region server crashed.
> >
> > I dug it and found the flow control does not work when write thread is
> > normal.
> >
> >
> > // If we crossed the chunk threshold, wait for more space to be available
> > synchronized (dataAvailable) {
> > while (totalBuffered > maxHeapUsage && thrown == null) {
> > LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> > waiting for IO threads...");
> > dataAvailable.wait(3000);
> > }
> > dataAvailable.notifyAll();
> > }
> >
> > If the code is below. It seems better.
> >
> > // If we crossed the chunk threshold, wait for more space to be available
> > synchronized (dataAvailable) {
> > while (totalBuffered > maxHeapUsage) {
> > LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> > waiting for IO threads...");
> > dataAvailable.wait(3000);
> > }
> > dataAvailable.notifyAll();
> > }
> >
> >
> >
> >
>
Re: Hmaster crashes caused by splitting log.
Posted by Gaojinchao <ga...@huawei.com>.
Thanks, You are right. :)
I will test it.
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年6月23日 11:00
收件人: dev@hbase.apache.org
主题: Re: Hmaster crashes caused by splitting log.
I guess you have seen this call directly below the following code snippet:
checkForErrors();
I think checking thrown should be kept in the condition.
I assume we can lift the above call before dataAvailable.notifyAll()
My 2 cents.
On Wed, Jun 22, 2011 at 7:00 PM, Gaojinchao <ga...@huawei.com> wrote:
>
> Because Master usually<javascript:void(0)> uses little memory. So its
> memory is 4G.
>
> DFS block is 256M and hbase.regionserver.maxlogs is 32. One region server
> can save max 8G Hlog.
>
> In my performance cluster(0.90.3), The Hmaster memory from 100 M up to 4G
> when one region server crashed.
>
> I dug it and found the flow control does not work when write thread is
> normal.
>
>
> // If we crossed the chunk threshold, wait for more space to be available
> synchronized (dataAvailable) {
> while (totalBuffered > maxHeapUsage && thrown == null) {
> LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> waiting for IO threads...");
> dataAvailable.wait(3000);
> }
> dataAvailable.notifyAll();
> }
>
> If the code is below. It seems better.
>
> // If we crossed the chunk threshold, wait for more space to be available
> synchronized (dataAvailable) {
> while (totalBuffered > maxHeapUsage) {
> LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> waiting for IO threads...");
> dataAvailable.wait(3000);
> }
> dataAvailable.notifyAll();
> }
>
>
>
>
Re: Hmaster crashes caused by splitting log.
Posted by Ted Yu <yu...@gmail.com>.
I guess you have seen this call directly below the following code snippet:
checkForErrors();
I think checking thrown should be kept in the condition.
I assume we can lift the above call before dataAvailable.notifyAll()
My 2 cents.
On Wed, Jun 22, 2011 at 7:00 PM, Gaojinchao <ga...@huawei.com> wrote:
>
> Because Master usually<javascript:void(0)> uses little memory. So its
> memory is 4G.
>
> DFS block is 256M and hbase.regionserver.maxlogs is 32. One region server
> can save max 8G Hlog.
>
> In my performance cluster(0.90.3), The Hmaster memory from 100 M up to 4G
> when one region server crashed.
>
> I dug it and found the flow control does not work when write thread is
> normal.
>
>
> // If we crossed the chunk threshold, wait for more space to be available
> synchronized (dataAvailable) {
> while (totalBuffered > maxHeapUsage && thrown == null) {
> LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> waiting for IO threads...");
> dataAvailable.wait(3000);
> }
> dataAvailable.notifyAll();
> }
>
> If the code is below. It seems better.
>
> // If we crossed the chunk threshold, wait for more space to be available
> synchronized (dataAvailable) {
> while (totalBuffered > maxHeapUsage) {
> LOG.debug("Used " + totalBuffered + " bytes of buffered edits,
> waiting for IO threads...");
> dataAvailable.wait(3000);
> }
> dataAvailable.notifyAll();
> }
>
>
>
>