You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by jingych <ji...@neusoft.com> on 2014/05/22 07:20:06 UTC
CDH5 hbase client outofmemory
Hello, everyone!
I found the CDH5 hbase client swallowed the Outofmemory exception.
It didn't throw out, cause the program couldn't make the correct response to process the OOM.
Is't good to catch the client OOM? Why?
The CDH5 hbase client process the throwable code:
private boolean manageError(int originalIndex, Row row, boolean canRetry,
Throwable throwable, HRegionLocation location) {
if (canRetry && throwable != null && throwable instanceof DoNotRetryIOException) {
canRetry = false;
}
byte[] region = null;
if (canRetry && callback != null) {
region = location == null ? null : location.getRegionInfo().getEncodedNameAsBytes();
canRetry = callback.retriableFailure(originalIndex, row, region, throwable);
}
if (!canRetry) {
if (callback != null) {
if (region == null && location != null) {
region = location.getRegionInfo().getEncodedNameAsBytes();
}
callback.failure(originalIndex, region, row, throwable);
}
errors.add(throwable, row, location);
this.hasError.set(true);
}
return canRetry;
}
This will treat the OOM as the canRetry exception.
Thanks!
Best Regards!
jingych
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: CDH5 hbase client outofmemory
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
postimage.org is also an option, etc.
2014-05-22 22:59 GMT-04:00 Ted Yu <yu...@gmail.com>:
> There're many options.
>
> https://*imgur*.com/ is one.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 7:53 PM, jingych <ji...@neusoft.com> wrote:
>
> > Sorry. This is a image. I do not know where can i upload it.
> >
> > Could you please give me some suggestions.
> >
> >
> >
> >
> > jingych
> >
> > From: Ted Yu
> > Date: 2014-05-23 10:45
> > To: user@hbase.apache.org
> > CC: Jean-Marc Spaggiari; user
> > Subject: Re: CDH5 hbase client outofmemory
> > The snapshot didn't go through.
> >
> > Can you put it on some website and give us the URL ?
> >
> > Cheers
> >
> > On May 22, 2014, at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> >
> > > Hi, JM & everyone!
> > >
> > > Thanks for your reply!
> > >
> > > My java client, use the HTable#put(Put) method with 2M buffer to commit
> > rows.
> > > While the client running for a while, the jvisualvm shows the htable
> > threads increasing and the heap size increased too.
> > > Here is the snapshot:
> > >
> > >
> > > i research the implementation of the HTable#put(Put).
> > > I found the doPut operation is the async, and when one row success,
> this
> > will return.
> > >
> > > So, I guess when the HBase server is slower to response the put
> request.
> > > The hbase client will accumulate so many rows.
> > > And on the other hand, when the put operation failed, the hbase client
> > will retry with the recursion submit method.
> > >
> > > I think the above two is the reason caused the OOME.
> > >
> > >
> > > PS:
> > > What do you mean It's not cdh5 specific? I found it from the website
> > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > > Where can i get the CDH5 source code ?
> > > CDH5 means hbase 0.96.2?
> > > I'm confused with the CDH5, Apache, Phoenix ...
> > > What's the relationship among them.
> > >
> > > Thanks a lot!
> > >
> > > Best regards!
> > >
> > > Jingych
> > >
> > > From: Jean-Marc Spaggiari
> > > Date: 2014-05-23 09:33
> > > To: user; jingych
> > > Subject: Re: CDH5 hbase client outofmemory
> > > Hi Jingych,
> > >
> > > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> > details on you OOME? Did you figured why it occured?
> > >
> > > JM
> > >
> > >
> > > 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> > >> Hello, everyone!
> > >>
> > >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> > >>
> > >> It didn't throw out, cause the program couldn't make the correct
> > response to process the OOM.
> > >>
> > >> Is't good to catch the client OOM? Why?
> > >>
> > >> The CDH5 hbase client process the throwable code:
> > >> private boolean manageError(int originalIndex, Row row, boolean
> > canRetry,
> > >> Throwable throwable, HRegionLocation
> > location) {
> > >> if (canRetry && throwable != null && throwable instanceof
> > DoNotRetryIOException) {
> > >> canRetry = false;
> > >> }
> > >>
> > >> byte[] region = null;
> > >> if (canRetry && callback != null) {
> > >> region = location == null ? null :
> > location.getRegionInfo().getEncodedNameAsBytes();
> > >> canRetry = callback.retriableFailure(originalIndex, row, region,
> > throwable);
> > >> }
> > >>
> > >> if (!canRetry) {
> > >> if (callback != null) {
> > >> if (region == null && location != null) {
> > >> region = location.getRegionInfo().getEncodedNameAsBytes();
> > >> }
> > >> callback.failure(originalIndex, region, row, throwable);
> > >> }
> > >> errors.add(throwable, row, location);
> > >> this.hasError.set(true);
> > >> }
> > >>
> > >> return canRetry;
> > >> }
> > >>
> > >> This will treat the OOM as the canRetry exception.
> > >>
> > >> Thanks!
> > >>
> > >> Best Regards!
> > >>
> > >>
> > >>
> > >>
> > >> jingych
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >> Confidentiality Notice: The information contained in this e-mail and
> > any accompanying attachment(s)
> > >> is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader of this communication is
> > >> not the intended recipient, unauthorized use, forwarding, printing,
> > storing, disclosure or copying
> > >> is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > >> immediately notify the sender by return e-mail, and delete the
> original
> > message and all copies from
> > >> your system. Thank you.
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > > Confidentiality Notice: The information contained in this e-mail and
> any
> > accompanying attachment(s)
> > > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader of this communication is
> > > not the intended recipient, unauthorized use, forwarding, printing,
> > storing, disclosure or copying
> > > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > > your system. Thank you.
> > >
> >
> ---------------------------------------------------------------------------------------------------
> >
> >
> ---------------------------------------------------------------------------------------------------
> > Confidentiality Notice: The information contained in this e-mail and any
> > accompanying attachment(s)
> > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader
> > of this communication is
> > not the intended recipient, unauthorized use, forwarding, printing,
> > storing, disclosure or copying
> > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > your system. Thank you.
> >
> >
> ---------------------------------------------------------------------------------------------------
> >
>
Re: Re: CDH5 hbase client outofmemory
Posted by jingych <ji...@neusoft.com>.
Thanks!
But we have the security policy, I can't access these website %>_<%
I try some, but doesn't work, i give up ╮(╯▽╰)╭
The post snapshot just monitor the heap size and threads.
It shows that：
The heap size is 512M.
While the client is running, the first 10min, the heap size is stable between the 150M and 300M. the gc works fine.
But when the client running about 12min, the heap size is reaching the warning line, 512M, and the gc is heavy-load.
In the meantime, the active thread size is increased to 125 from 100.
Best Regards!
Jingych
From: Ted Yu
Date: 2014-05-23 10:59
To: user@hbase.apache.org; jingych
Subject: Re: Re: CDH5 hbase client outofmemory
There're many options.
https://imgur.com/ is one.
Cheers
On Thu, May 22, 2014 at 7:53 PM, jingych <ji...@neusoft.com> wrote:
Sorry. This is a image. I do not know where can i upload it.
Could you please give me some suggestions.
jingych
From: Ted Yu
Date: 2014-05-23 10:45
To: user@hbase.apache.org
CC: Jean-Marc Spaggiari; user
Subject: Re: CDH5 hbase client outofmemory
The snapshot didn't go through.
Can you put it on some website and give us the URL ?
Cheers
On May 22, 2014, at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> Hi, JM & everyone!
>
> Thanks for your reply!
>
> My java client, use the HTable#put(Put) method with 2M buffer to commit rows.
> While the client running for a while, the jvisualvm shows the htable threads increasing and the heap size increased too.
> Here is the snapshot:
>
>
> i research the implementation of the HTable#put(Put).
> I found the doPut operation is the async, and when one row success, this will return.
>
> So, I guess when the HBase server is slower to response the put request.
> The hbase client will accumulate so many rows.
> And on the other hand, when the put operation failed, the hbase client will retry with the recursion submit method.
>
> I think the above two is the reason caused the OOME.
>
>
> PS:
> What do you mean It's not cdh5 specific? I found it from the website http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> Where can i get the CDH5 source code ?
> CDH5 means hbase 0.96.2?
> I'm confused with the CDH5, Apache, Phoenix ...
> What's the relationship among them.
>
> Thanks a lot!
>
> Best regards!
>
> Jingych
>
> From: Jean-Marc Spaggiari
> Date: 2014-05-23 09:33
> To: user; jingych
> Subject: Re: CDH5 hbase client outofmemory
> Hi Jingych,
>
> This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details on you OOME? Did you figured why it occured?
>
> JM
>
>
> 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
>> Hello, everyone!
>>
>> I found the CDH5 hbase client swallowed the Outofmemory exception.
>>
>> It didn't throw out, cause the program couldn't make the correct response to process the OOM.
>>
>> Is't good to catch the client OOM? Why?
>>
>> The CDH5 hbase client process the throwable code:
>> private boolean manageError(int originalIndex, Row row, boolean canRetry,
>> Throwable throwable, HRegionLocation location) {
>> if (canRetry && throwable != null && throwable instanceof DoNotRetryIOException) {
>> canRetry = false;
>> }
>>
>> byte[] region = null;
>> if (canRetry && callback != null) {
>> region = location == null ? null : location.getRegionInfo().getEncodedNameAsBytes();
>> canRetry = callback.retriableFailure(originalIndex, row, region, throwable);
>> }
>>
>> if (!canRetry) {
>> if (callback != null) {
>> if (region == null && location != null) {
>> region = location.getRegionInfo().getEncodedNameAsBytes();
>> }
>> callback.failure(originalIndex, region, row, throwable);
>> }
>> errors.add(throwable, row, location);
>> this.hasError.set(true);
>> }
>>
>> return canRetry;
>> }
>>
>> This will treat the OOM as the canRetry exception.
>>
>> Thanks!
>>
>> Best Regards!
>>
>>
>>
>>
>> jingych
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
>> immediately notify the sender by return e-mail, and delete the original message and all copies from
>> your system. Thank you.
>> ---------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
> is intended only for the use of the intended recipient and may be confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
> immediately notify the sender by return e-mail, and delete the original message and all copies from
> your system. Thank you.
> ---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: CDH5 hbase client outofmemory
Posted by Ted Yu <yu...@gmail.com>.
There're many options.
https://*imgur*.com/ is one.
Cheers
On Thu, May 22, 2014 at 7:53 PM, jingych <ji...@neusoft.com> wrote:
> Sorry. This is a image. I do not know where can i upload it.
>
> Could you please give me some suggestions.
>
>
>
>
> jingych
>
> From: Ted Yu
> Date: 2014-05-23 10:45
> To: user@hbase.apache.org
> CC: Jean-Marc Spaggiari; user
> Subject: Re: CDH5 hbase client outofmemory
> The snapshot didn't go through.
>
> Can you put it on some website and give us the URL ?
>
> Cheers
>
> On May 22, 2014, at 7:38 PM, jingych <ji...@neusoft.com> wrote:
>
> > Hi, JM & everyone!
> >
> > Thanks for your reply!
> >
> > My java client, use the HTable#put(Put) method with 2M buffer to commit
> rows.
> > While the client running for a while, the jvisualvm shows the htable
> threads increasing and the heap size increased too.
> > Here is the snapshot:
> >
> >
> > i research the implementation of the HTable#put(Put).
> > I found the doPut operation is the async, and when one row success, this
> will return.
> >
> > So, I guess when the HBase server is slower to response the put request.
> > The hbase client will accumulate so many rows.
> > And on the other hand, when the put operation failed, the hbase client
> will retry with the recursion submit method.
> >
> > I think the above two is the reason caused the OOME.
> >
> >
> > PS:
> > What do you mean It's not cdh5 specific? I found it from the website
> http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > Where can i get the CDH5 source code ?
> > CDH5 means hbase 0.96.2?
> > I'm confused with the CDH5, Apache, Phoenix ...
> > What's the relationship among them.
> >
> > Thanks a lot!
> >
> > Best regards!
> >
> > Jingych
> >
> > From: Jean-Marc Spaggiari
> > Date: 2014-05-23 09:33
> > To: user; jingych
> > Subject: Re: CDH5 hbase client outofmemory
> > Hi Jingych,
> >
> > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> details on you OOME? Did you figured why it occured?
> >
> > JM
> >
> >
> > 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> >> Hello, everyone!
> >>
> >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> >>
> >> It didn't throw out, cause the program couldn't make the correct
> response to process the OOM.
> >>
> >> Is't good to catch the client OOM? Why?
> >>
> >> The CDH5 hbase client process the throwable code:
> >> private boolean manageError(int originalIndex, Row row, boolean
> canRetry,
> >> Throwable throwable, HRegionLocation
> location) {
> >> if (canRetry && throwable != null && throwable instanceof
> DoNotRetryIOException) {
> >> canRetry = false;
> >> }
> >>
> >> byte[] region = null;
> >> if (canRetry && callback != null) {
> >> region = location == null ? null :
> location.getRegionInfo().getEncodedNameAsBytes();
> >> canRetry = callback.retriableFailure(originalIndex, row, region,
> throwable);
> >> }
> >>
> >> if (!canRetry) {
> >> if (callback != null) {
> >> if (region == null && location != null) {
> >> region = location.getRegionInfo().getEncodedNameAsBytes();
> >> }
> >> callback.failure(originalIndex, region, row, throwable);
> >> }
> >> errors.add(throwable, row, location);
> >> this.hasError.set(true);
> >> }
> >>
> >> return canRetry;
> >> }
> >>
> >> This will treat the OOM as the canRetry exception.
> >>
> >> Thanks!
> >>
> >> Best Regards!
> >>
> >>
> >>
> >>
> >> jingych
> >>
> ---------------------------------------------------------------------------------------------------
> >> Confidentiality Notice: The information contained in this e-mail and
> any accompanying attachment(s)
> >> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader of this communication is
> >> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> >> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> >> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> >> your system. Thank you.
> >>
> ---------------------------------------------------------------------------------------------------
> >
> >
> ---------------------------------------------------------------------------------------------------
> > Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> > is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader of this communication is
> > not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> > is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> > immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> > your system. Thank you.
> >
> ---------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
Re: Re: CDH5 hbase client outofmemory
Posted by jingych <ji...@neusoft.com>.
Sorry. This is a image. I do not know where can i upload it.
Could you please give me some suggestions.
jingych
From: Ted Yu
Date: 2014-05-23 10:45
To: user@hbase.apache.org
CC: Jean-Marc Spaggiari; user
Subject: Re: CDH5 hbase client outofmemory
The snapshot didn't go through.
Can you put it on some website and give us the URL ?
Cheers
On May 22, 2014, at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> Hi, JM & everyone!
>
> Thanks for your reply!
>
> My java client, use the HTable#put(Put) method with 2M buffer to commit rows.
> While the client running for a while, the jvisualvm shows the htable threads increasing and the heap size increased too.
> Here is the snapshot:
>
>
> i research the implementation of the HTable#put(Put).
> I found the doPut operation is the async, and when one row success, this will return.
>
> So, I guess when the HBase server is slower to response the put request.
> The hbase client will accumulate so many rows.
> And on the other hand, when the put operation failed, the hbase client will retry with the recursion submit method.
>
> I think the above two is the reason caused the OOME.
>
>
> PS:
> What do you mean It's not cdh5 specific? I found it from the website http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> Where can i get the CDH5 source code ?
> CDH5 means hbase 0.96.2?
> I'm confused with the CDH5, Apache, Phoenix ...
> What's the relationship among them.
>
> Thanks a lot!
>
> Best regards!
>
> Jingych
>
> From: Jean-Marc Spaggiari
> Date: 2014-05-23 09:33
> To: user; jingych
> Subject: Re: CDH5 hbase client outofmemory
> Hi Jingych,
>
> This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details on you OOME? Did you figured why it occured?
>
> JM
>
>
> 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
>> Hello, everyone!
>>
>> I found the CDH5 hbase client swallowed the Outofmemory exception.
>>
>> It didn't throw out, cause the program couldn't make the correct response to process the OOM.
>>
>> Is't good to catch the client OOM? Why?
>>
>> The CDH5 hbase client process the throwable code:
>> private boolean manageError(int originalIndex, Row row, boolean canRetry,
>> Throwable throwable, HRegionLocation location) {
>> if (canRetry && throwable != null && throwable instanceof DoNotRetryIOException) {
>> canRetry = false;
>> }
>>
>> byte[] region = null;
>> if (canRetry && callback != null) {
>> region = location == null ? null : location.getRegionInfo().getEncodedNameAsBytes();
>> canRetry = callback.retriableFailure(originalIndex, row, region, throwable);
>> }
>>
>> if (!canRetry) {
>> if (callback != null) {
>> if (region == null && location != null) {
>> region = location.getRegionInfo().getEncodedNameAsBytes();
>> }
>> callback.failure(originalIndex, region, row, throwable);
>> }
>> errors.add(throwable, row, location);
>> this.hasError.set(true);
>> }
>>
>> return canRetry;
>> }
>>
>> This will treat the OOM as the canRetry exception.
>>
>> Thanks!
>>
>> Best Regards!
>>
>>
>>
>>
>> jingych
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
>> immediately notify the sender by return e-mail, and delete the original message and all copies from
>> your system. Thank you.
>> ---------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
> is intended only for the use of the intended recipient and may be confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
> immediately notify the sender by return e-mail, and delete the original message and all copies from
> your system. Thank you.
> ---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: CDH5 hbase client outofmemory
Posted by Ted Yu <yu...@gmail.com>.
The snapshot didn't go through.
Can you put it on some website and give us the URL ?
Cheers
On May 22, 2014, at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> Hi, JM & everyone!
>
> Thanks for your reply!
>
> My java client, use the HTable#put(Put) method with 2M buffer to commit rows.
> While the client running for a while, the jvisualvm shows the htable threads increasing and the heap size increased too.
> Here is the snapshot:
>
>
> i research the implementation of the HTable#put(Put).
> I found the doPut operation is the async, and when one row success, this will return.
>
> So, I guess when the HBase server is slower to response the put request.
> The hbase client will accumulate so many rows.
> And on the other hand, when the put operation failed, the hbase client will retry with the recursion submit method.
>
> I think the above two is the reason caused the OOME.
>
>
> PS:
> What do you mean It's not cdh5 specific? I found it from the website http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> Where can i get the CDH5 source code ?
> CDH5 means hbase 0.96.2?
> I'm confused with the CDH5, Apache, Phoenix ...
> What's the relationship among them.
>
> Thanks a lot!
>
> Best regards!
>
> Jingych
>
> From: Jean-Marc Spaggiari
> Date: 2014-05-23 09:33
> To: user; jingych
> Subject: Re: CDH5 hbase client outofmemory
> Hi Jingych,
>
> This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details on you OOME? Did you figured why it occured?
>
> JM
>
>
> 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
>> Hello, everyone!
>>
>> I found the CDH5 hbase client swallowed the Outofmemory exception.
>>
>> It didn't throw out, cause the program couldn't make the correct response to process the OOM.
>>
>> Is't good to catch the client OOM? Why?
>>
>> The CDH5 hbase client process the throwable code:
>> private boolean manageError(int originalIndex, Row row, boolean canRetry,
>> Throwable throwable, HRegionLocation location) {
>> if (canRetry && throwable != null && throwable instanceof DoNotRetryIOException) {
>> canRetry = false;
>> }
>>
>> byte[] region = null;
>> if (canRetry && callback != null) {
>> region = location == null ? null : location.getRegionInfo().getEncodedNameAsBytes();
>> canRetry = callback.retriableFailure(originalIndex, row, region, throwable);
>> }
>>
>> if (!canRetry) {
>> if (callback != null) {
>> if (region == null && location != null) {
>> region = location.getRegionInfo().getEncodedNameAsBytes();
>> }
>> callback.failure(originalIndex, region, row, throwable);
>> }
>> errors.add(throwable, row, location);
>> this.hasError.set(true);
>> }
>>
>> return canRetry;
>> }
>>
>> This will treat the OOM as the canRetry exception.
>>
>> Thanks!
>>
>> Best Regards!
>>
>>
>>
>>
>> jingych
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
>> immediately notify the sender by return e-mail, and delete the original message and all copies from
>> your system. Thank you.
>> ---------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
> is intended only for the use of the intended recipient and may be confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
> not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this communication in error,please
> immediately notify the sender by return e-mail, and delete the original message and all copies from
> your system. Thank you.
> ---------------------------------------------------------------------------------------------------
Re: Re: CDH5 hbase client outofmemory
Posted by Ted Yu <yu...@gmail.com>.
bq. Where can i post the bug list?
You can find HBase JIRAs here:
https://issues.apache.org/jira/browse/HBASE/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel
Re: Re: CDH5 hbase client outofmemory
Posted by jingych <ji...@neusoft.com>.
Hi, Anoop & Nicolas!
Sorry for reply you so late!
I'm a newbie in the hadoop community. Where can i post the bug list?
And now I found that the HTable#flushCommit method is the sync operation.
So my solution is counting the rows, if the rows cached by client beyond the threadhold, i flush the cache.
And now the problem seemingly solved.O(∩_∩)O~
Best Regards!
jingych
From: Anoop John
Date: 2014-05-23 18:23
To: user@hbase.apache.org
CC: jingych
Subject: Re: Re: CDH5 hbase client outofmemory
>>I found the doPut operation is the async, and when one row success, this
will return.
>>So, I guess when the HBase server is slower to response the put request.
>>The hbase client will accumulate so many rows.
That also sounds like a bug. Can add this also in the bug u r going to
raise.
-Anoop-
On Fri, May 23, 2014 at 2:11 PM, Nicolas Liochon <nk...@gmail.com> wrote:
> This can be considered as a bug. We're not going to survive long if there
> is an OOM, so retrying is not very useful... You can create a jira for this
> (https://issues.apache.org/jira/browse/HBASE), and submit a patch if
> possible..
>
> For the root cause, if you don't have much memory, you can lower the buffer
> size (hbase.client.write.buffer) or the number of simultaneous tasks (
> hbase.client.max.total.tasks)
>
>
> On Fri, May 23, 2014 at 5:44 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > w.r.t. your questions toward the bottom of your email:
> > CDH HBase is one distribution of Apache HBase. As you can see from the
> url,
> > it is based on 0.96.1.1
> > HDP is another distribution.
> >
> > There're many projects under Apache Software Foundation, such as YARN,
> > HDFS, HBase, Phoenix.
> > You can get more information on respective project through their JIRAs,
> > mailing lists, source repository, etc.
> >
> > You can get 0.96 source code using the following command:
> > git clone -b 0.96 https://git-wip-us.apache.org/repos/asf/hbase.git<local
> > directory name>
> >
> > You can also get more help from CDH mailing list.
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> >
> > > Hi, JM & everyone!
> > >
> > > Thanks for your reply!
> > >
> > > My java client, use the HTable#put(Put) method with 2M buffer to commit
> > > rows.
> > > While the client running for a while, the jvisualvm shows the htable
> > > threads increasing and the heap size increased too.
> > > Here is the snapshot:
> > >
> > > i research the implementation of the HTable#put(Put).
> > > I found the doPut operation is the async, and when one row success,
> this
> > > will return.
> > >
> > > So, I guess when the HBase server is slower to response the put
> request.
> > > The hbase client will accumulate so many rows.
> > > And on the other hand, when the put operation failed, the hbase client
> > > will retry with the recursion <http://dict.youdao.com/w/recursion/>
> > submit
> > > method.
> > >
> > > I think the above two is the reason caused the OOME.
> > >
> > >
> > > PS:
> > > What do you mean It's not cdh5 specific? I found it from the website
> > > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > > Where can i get the CDH5 source code ?
> > > CDH5 means hbase 0.96.2?
> > > I'm confused with the CDH5, Apache, Phoenix ...
> > > What's the relationship among them.
> > >
> > > Thanks a lot!
> > >
> > > Best regards!
> > >
> > > ------------------------------
> > >
> > > Jingych
> > >
> > > *From:* Jean-Marc Spaggiari <je...@spaggiari.org>
> > > *Date:* 2014-05-23 09:33
> > > *To:* user <us...@hbase.apache.org>; jingych <ji...@neusoft.com>
> > > *Subject:* Re: CDH5 hbase client outofmemory
> > > Hi Jingych,
> > >
> > > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> > details
> > > on you OOME? Did you figured why it occured?
> > >
> > > JM
> > >
> > >
> > > 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> > >
> > > Hello, everyone!
> > >>
> > >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> > >>
> > >> It didn't throw out, cause the program couldn't make the correct
> > response
> > >> to process the OOM.
> > >>
> > >> Is't good to catch the client OOM? Why?
> > >>
> > >> The CDH5 hbase client process the throwable code:
> > >> private boolean manageError(int originalIndex, Row row, boolean
> > canRetry,
> > >> Throwable throwable, HRegionLocation
> > >> location) {
> > >> if (canRetry && throwable != null && throwable instanceof
> > >> DoNotRetryIOException) {
> > >> canRetry = false;
> > >> }
> > >>
> > >> byte[] region = null;
> > >> if (canRetry && callback != null) {
> > >> region = location == null ? null :
> > >> location.getRegionInfo().getEncodedNameAsBytes();
> > >> canRetry = callback.retriableFailure(originalIndex, row, region,
> > >> throwable);
> > >> }
> > >>
> > >> if (!canRetry) {
> > >> if (callback != null) {
> > >> if (region == null && location != null) {
> > >> region = location.getRegionInfo().getEncodedNameAsBytes();
> > >> }
> > >> callback.failure(originalIndex, region, row, throwable);
> > >> }
> > >> errors.add(throwable, row, location);
> > >> this.hasError.set(true);
> > >> }
> > >>
> > >> return canRetry;
> > >> }
> > >>
> > >> This will treat the OOM as the canRetry exception.
> > >>
> > >> Thanks!
> > >>
> > >> Best Regards!
> > >>
> > >>
> > >>
> > >>
> > >> jingych
> > >>
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >> Confidentiality Notice: The information contained in this e-mail and
> any
> > >> accompanying attachment(s)
> > >> is intended only for the use of the intended recipient and may be
> > >> confidential and/or privileged of
> > >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > >> reader of this communication is
> > >> not the intended recipient, unauthorized use, forwarding, printing,
> > >> storing, disclosure or copying
> > >> is strictly prohibited, and may be unlawful.If you have received this
> > >> communication in error,please
> > >> immediately notify the sender by return e-mail, and delete the
> original
> > >> message and all copies from
> > >> your system. Thank you.
> > >>
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >>
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > > Confidentiality Notice: The information contained in this e-mail and
> any
> > > accompanying attachment(s)
> > > is intended only for the use of the intended recipient and may be
> > > confidential and/or privileged of
> > > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader
> > > of this communication is
> > > not the intended recipient, unauthorized use, forwarding, printing,
> > > storing, disclosure or copying
> > > is strictly prohibited, and may be unlawful.If you have received this
> > > communication in error,please
> > > immediately notify the sender by return e-mail, and delete the original
> > > message and all copies from
> > > your system. Thank you.
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > >
> >
>
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Re: CDH5 hbase client outofmemory
Posted by Anoop John <an...@gmail.com>.
>>I found the doPut operation is the async, and when one row success, this
will return.
>>So, I guess when the HBase server is slower to response the put request.
>>The hbase client will accumulate so many rows.
That also sounds like a bug. Can add this also in the bug u r going to
raise.
-Anoop-
On Fri, May 23, 2014 at 2:11 PM, Nicolas Liochon <nk...@gmail.com> wrote:
> This can be considered as a bug. We're not going to survive long if there
> is an OOM, so retrying is not very useful... You can create a jira for this
> (https://issues.apache.org/jira/browse/HBASE), and submit a patch if
> possible..
>
> For the root cause, if you don't have much memory, you can lower the buffer
> size (hbase.client.write.buffer) or the number of simultaneous tasks (
> hbase.client.max.total.tasks)
>
>
> On Fri, May 23, 2014 at 5:44 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > w.r.t. your questions toward the bottom of your email:
> > CDH HBase is one distribution of Apache HBase. As you can see from the
> url,
> > it is based on 0.96.1.1
> > HDP is another distribution.
> >
> > There're many projects under Apache Software Foundation, such as YARN,
> > HDFS, HBase, Phoenix.
> > You can get more information on respective project through their JIRAs,
> > mailing lists, source repository, etc.
> >
> > You can get 0.96 source code using the following command:
> > git clone -b 0.96 https://git-wip-us.apache.org/repos/asf/hbase.git<local
> > directory name>
> >
> > You can also get more help from CDH mailing list.
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> >
> > > Hi, JM & everyone!
> > >
> > > Thanks for your reply!
> > >
> > > My java client, use the HTable#put(Put) method with 2M buffer to commit
> > > rows.
> > > While the client running for a while, the jvisualvm shows the htable
> > > threads increasing and the heap size increased too.
> > > Here is the snapshot:
> > >
> > > i research the implementation of the HTable#put(Put).
> > > I found the doPut operation is the async, and when one row success,
> this
> > > will return.
> > >
> > > So, I guess when the HBase server is slower to response the put
> request.
> > > The hbase client will accumulate so many rows.
> > > And on the other hand, when the put operation failed, the hbase client
> > > will retry with the recursion <http://dict.youdao.com/w/recursion/>
> > submit
> > > method.
> > >
> > > I think the above two is the reason caused the OOME.
> > >
> > >
> > > PS:
> > > What do you mean It's not cdh5 specific? I found it from the website
> > > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > > Where can i get the CDH5 source code ?
> > > CDH5 means hbase 0.96.2?
> > > I'm confused with the CDH5, Apache, Phoenix ...
> > > What's the relationship among them.
> > >
> > > Thanks a lot!
> > >
> > > Best regards!
> > >
> > > ------------------------------
> > >
> > > Jingych
> > >
> > > *From:* Jean-Marc Spaggiari <je...@spaggiari.org>
> > > *Date:* 2014-05-23 09:33
> > > *To:* user <us...@hbase.apache.org>; jingych <ji...@neusoft.com>
> > > *Subject:* Re: CDH5 hbase client outofmemory
> > > Hi Jingych,
> > >
> > > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> > details
> > > on you OOME? Did you figured why it occured?
> > >
> > > JM
> > >
> > >
> > > 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> > >
> > > Hello, everyone!
> > >>
> > >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> > >>
> > >> It didn't throw out, cause the program couldn't make the correct
> > response
> > >> to process the OOM.
> > >>
> > >> Is't good to catch the client OOM? Why?
> > >>
> > >> The CDH5 hbase client process the throwable code:
> > >> private boolean manageError(int originalIndex, Row row, boolean
> > canRetry,
> > >> Throwable throwable, HRegionLocation
> > >> location) {
> > >> if (canRetry && throwable != null && throwable instanceof
> > >> DoNotRetryIOException) {
> > >> canRetry = false;
> > >> }
> > >>
> > >> byte[] region = null;
> > >> if (canRetry && callback != null) {
> > >> region = location == null ? null :
> > >> location.getRegionInfo().getEncodedNameAsBytes();
> > >> canRetry = callback.retriableFailure(originalIndex, row, region,
> > >> throwable);
> > >> }
> > >>
> > >> if (!canRetry) {
> > >> if (callback != null) {
> > >> if (region == null && location != null) {
> > >> region = location.getRegionInfo().getEncodedNameAsBytes();
> > >> }
> > >> callback.failure(originalIndex, region, row, throwable);
> > >> }
> > >> errors.add(throwable, row, location);
> > >> this.hasError.set(true);
> > >> }
> > >>
> > >> return canRetry;
> > >> }
> > >>
> > >> This will treat the OOM as the canRetry exception.
> > >>
> > >> Thanks!
> > >>
> > >> Best Regards!
> > >>
> > >>
> > >>
> > >>
> > >> jingych
> > >>
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >> Confidentiality Notice: The information contained in this e-mail and
> any
> > >> accompanying attachment(s)
> > >> is intended only for the use of the intended recipient and may be
> > >> confidential and/or privileged of
> > >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > >> reader of this communication is
> > >> not the intended recipient, unauthorized use, forwarding, printing,
> > >> storing, disclosure or copying
> > >> is strictly prohibited, and may be unlawful.If you have received this
> > >> communication in error,please
> > >> immediately notify the sender by return e-mail, and delete the
> original
> > >> message and all copies from
> > >> your system. Thank you.
> > >>
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >>
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > > Confidentiality Notice: The information contained in this e-mail and
> any
> > > accompanying attachment(s)
> > > is intended only for the use of the intended recipient and may be
> > > confidential and/or privileged of
> > > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader
> > > of this communication is
> > > not the intended recipient, unauthorized use, forwarding, printing,
> > > storing, disclosure or copying
> > > is strictly prohibited, and may be unlawful.If you have received this
> > > communication in error,please
> > > immediately notify the sender by return e-mail, and delete the original
> > > message and all copies from
> > > your system. Thank you.
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > >
> >
>
Re: Re: CDH5 hbase client outofmemory
Posted by Nicolas Liochon <nk...@gmail.com>.
This can be considered as a bug. We're not going to survive long if there
is an OOM, so retrying is not very useful... You can create a jira for this
(https://issues.apache.org/jira/browse/HBASE), and submit a patch if
possible..
For the root cause, if you don't have much memory, you can lower the buffer
size (hbase.client.write.buffer) or the number of simultaneous tasks (
hbase.client.max.total.tasks)
On Fri, May 23, 2014 at 5:44 AM, Ted Yu <yu...@gmail.com> wrote:
> w.r.t. your questions toward the bottom of your email:
> CDH HBase is one distribution of Apache HBase. As you can see from the url,
> it is based on 0.96.1.1
> HDP is another distribution.
>
> There're many projects under Apache Software Foundation, such as YARN,
> HDFS, HBase, Phoenix.
> You can get more information on respective project through their JIRAs,
> mailing lists, source repository, etc.
>
> You can get 0.96 source code using the following command:
> git clone -b 0.96 https://git-wip-us.apache.org/repos/asf/hbase.git <local
> directory name>
>
> You can also get more help from CDH mailing list.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 7:38 PM, jingych <ji...@neusoft.com> wrote:
>
> > Hi, JM & everyone!
> >
> > Thanks for your reply!
> >
> > My java client, use the HTable#put(Put) method with 2M buffer to commit
> > rows.
> > While the client running for a while, the jvisualvm shows the htable
> > threads increasing and the heap size increased too.
> > Here is the snapshot:
> >
> > i research the implementation of the HTable#put(Put).
> > I found the doPut operation is the async, and when one row success, this
> > will return.
> >
> > So, I guess when the HBase server is slower to response the put request.
> > The hbase client will accumulate so many rows.
> > And on the other hand, when the put operation failed, the hbase client
> > will retry with the recursion <http://dict.youdao.com/w/recursion/>
> submit
> > method.
> >
> > I think the above two is the reason caused the OOME.
> >
> >
> > PS:
> > What do you mean It's not cdh5 specific? I found it from the website
> > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > Where can i get the CDH5 source code ?
> > CDH5 means hbase 0.96.2?
> > I'm confused with the CDH5, Apache, Phoenix ...
> > What's the relationship among them.
> >
> > Thanks a lot!
> >
> > Best regards!
> >
> > ------------------------------
> >
> > Jingych
> >
> > *From:* Jean-Marc Spaggiari <je...@spaggiari.org>
> > *Date:* 2014-05-23 09:33
> > *To:* user <us...@hbase.apache.org>; jingych <ji...@neusoft.com>
> > *Subject:* Re: CDH5 hbase client outofmemory
> > Hi Jingych,
> >
> > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> details
> > on you OOME? Did you figured why it occured?
> >
> > JM
> >
> >
> > 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> >
> > Hello, everyone!
> >>
> >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> >>
> >> It didn't throw out, cause the program couldn't make the correct
> response
> >> to process the OOM.
> >>
> >> Is't good to catch the client OOM? Why?
> >>
> >> The CDH5 hbase client process the throwable code:
> >> private boolean manageError(int originalIndex, Row row, boolean
> canRetry,
> >> Throwable throwable, HRegionLocation
> >> location) {
> >> if (canRetry && throwable != null && throwable instanceof
> >> DoNotRetryIOException) {
> >> canRetry = false;
> >> }
> >>
> >> byte[] region = null;
> >> if (canRetry && callback != null) {
> >> region = location == null ? null :
> >> location.getRegionInfo().getEncodedNameAsBytes();
> >> canRetry = callback.retriableFailure(originalIndex, row, region,
> >> throwable);
> >> }
> >>
> >> if (!canRetry) {
> >> if (callback != null) {
> >> if (region == null && location != null) {
> >> region = location.getRegionInfo().getEncodedNameAsBytes();
> >> }
> >> callback.failure(originalIndex, region, row, throwable);
> >> }
> >> errors.add(throwable, row, location);
> >> this.hasError.set(true);
> >> }
> >>
> >> return canRetry;
> >> }
> >>
> >> This will treat the OOM as the canRetry exception.
> >>
> >> Thanks!
> >>
> >> Best Regards!
> >>
> >>
> >>
> >>
> >> jingych
> >>
> >>
> ---------------------------------------------------------------------------------------------------
> >> Confidentiality Notice: The information contained in this e-mail and any
> >> accompanying attachment(s)
> >> is intended only for the use of the intended recipient and may be
> >> confidential and/or privileged of
> >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> >> reader of this communication is
> >> not the intended recipient, unauthorized use, forwarding, printing,
> >> storing, disclosure or copying
> >> is strictly prohibited, and may be unlawful.If you have received this
> >> communication in error,please
> >> immediately notify the sender by return e-mail, and delete the original
> >> message and all copies from
> >> your system. Thank you.
> >>
> >>
> ---------------------------------------------------------------------------------------------------
> >>
> >
> >
> >
> ---------------------------------------------------------------------------------------------------
> > Confidentiality Notice: The information contained in this e-mail and any
> > accompanying attachment(s)
> > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader
> > of this communication is
> > not the intended recipient, unauthorized use, forwarding, printing,
> > storing, disclosure or copying
> > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > your system. Thank you.
> >
> >
> ---------------------------------------------------------------------------------------------------
> >
>
Re: Re: CDH5 hbase client outofmemory
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. your questions toward the bottom of your email:
CDH HBase is one distribution of Apache HBase. As you can see from the url,
it is based on 0.96.1.1
HDP is another distribution.
There're many projects under Apache Software Foundation, such as YARN,
HDFS, HBase, Phoenix.
You can get more information on respective project through their JIRAs,
mailing lists, source repository, etc.
You can get 0.96 source code using the following command:
git clone -b 0.96 https://git-wip-us.apache.org/repos/asf/hbase.git <local
directory name>
You can also get more help from CDH mailing list.
Cheers
On Thu, May 22, 2014 at 7:38 PM, jingych <ji...@neusoft.com> wrote:
> Hi, JM & everyone!
>
> Thanks for your reply!
>
> My java client, use the HTable#put(Put) method with 2M buffer to commit
> rows.
> While the client running for a while, the jvisualvm shows the htable
> threads increasing and the heap size increased too.
> Here is the snapshot:
>
> i research the implementation of the HTable#put(Put).
> I found the doPut operation is the async, and when one row success, this
> will return.
>
> So, I guess when the HBase server is slower to response the put request.
> The hbase client will accumulate so many rows.
> And on the other hand, when the put operation failed, the hbase client
> will retry with the recursion <http://dict.youdao.com/w/recursion/> submit
> method.
>
> I think the above two is the reason caused the OOME.
>
>
> PS:
> What do you mean It's not cdh5 specific? I found it from the website
> http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> Where can i get the CDH5 source code ?
> CDH5 means hbase 0.96.2?
> I'm confused with the CDH5, Apache, Phoenix ...
> What's the relationship among them.
>
> Thanks a lot!
>
> Best regards!
>
> ------------------------------
>
> Jingych
>
> *From:* Jean-Marc Spaggiari <je...@spaggiari.org>
> *Date:* 2014-05-23 09:33
> *To:* user <us...@hbase.apache.org>; jingych <ji...@neusoft.com>
> *Subject:* Re: CDH5 hbase client outofmemory
> Hi Jingych,
>
> This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details
> on you OOME? Did you figured why it occured?
>
> JM
>
>
> 2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
>
> Hello, everyone!
>>
>> I found the CDH5 hbase client swallowed the Outofmemory exception.
>>
>> It didn't throw out, cause the program couldn't make the correct response
>> to process the OOM.
>>
>> Is't good to catch the client OOM? Why?
>>
>> The CDH5 hbase client process the throwable code:
>> private boolean manageError(int originalIndex, Row row, boolean canRetry,
>> Throwable throwable, HRegionLocation
>> location) {
>> if (canRetry && throwable != null && throwable instanceof
>> DoNotRetryIOException) {
>> canRetry = false;
>> }
>>
>> byte[] region = null;
>> if (canRetry && callback != null) {
>> region = location == null ? null :
>> location.getRegionInfo().getEncodedNameAsBytes();
>> canRetry = callback.retriableFailure(originalIndex, row, region,
>> throwable);
>> }
>>
>> if (!canRetry) {
>> if (callback != null) {
>> if (region == null && location != null) {
>> region = location.getRegionInfo().getEncodedNameAsBytes();
>> }
>> callback.failure(originalIndex, region, row, throwable);
>> }
>> errors.add(throwable, row, location);
>> this.hasError.set(true);
>> }
>>
>> return canRetry;
>> }
>>
>> This will treat the OOM as the canRetry exception.
>>
>> Thanks!
>>
>> Best Regards!
>>
>>
>>
>>
>> jingych
>>
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any
>> accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be
>> confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>> reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing,
>> storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this
>> communication in error,please
>> immediately notify the sender by return e-mail, and delete the original
>> message and all copies from
>> your system. Thank you.
>>
>> ---------------------------------------------------------------------------------------------------
>>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
Re: Re: CDH5 hbase client outofmemory
Posted by jingych <ji...@neusoft.com>.
Hi, JM & everyone!
Thanks for your reply!
My java client, use the HTable#put(Put) method with 2M buffer to commit rows.
While the client running for a while, the jvisualvm shows the htable threads increasing and the heap size increased too.
Here is the snapshot:
i research the implementation of the HTable#put(Put).
I found the doPut operation is the async, and when one row success, this will return.
So, I guess when the HBase server is slower to response the put request.
The hbase client will accumulate so many rows.
And on the other hand, when the put operation failed, the hbase client will retry with the recursion submit method.
I think the above two is the reason caused the OOME.
PS:
What do you mean It's not cdh5 specific? I found it from the website http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
Where can i get the CDH5 source code ?
CDH5 means hbase 0.96.2?
I'm confused with the CDH5, Apache, Phoenix ...
What's the relationship among them.
Thanks a lot!
Best regards!
Jingych
From: Jean-Marc Spaggiari
Date: 2014-05-23 09:33
To: user; jingych
Subject: Re: CDH5 hbase client outofmemory
Hi Jingych,
This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details on you OOME? Did you figured why it occured?
JM
2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
Hello, everyone!
I found the CDH5 hbase client swallowed the Outofmemory exception.
It didn't throw out, cause the program couldn't make the correct response to process the OOM.
Is't good to catch the client OOM? Why?
The CDH5 hbase client process the throwable code:
private boolean manageError(int originalIndex, Row row, boolean canRetry,
Throwable throwable, HRegionLocation location) {
if (canRetry && throwable != null && throwable instanceof DoNotRetryIOException) {
canRetry = false;
}
byte[] region = null;
if (canRetry && callback != null) {
region = location == null ? null : location.getRegionInfo().getEncodedNameAsBytes();
canRetry = callback.retriableFailure(originalIndex, row, region, throwable);
}
if (!canRetry) {
if (callback != null) {
if (region == null && location != null) {
region = location.getRegionInfo().getEncodedNameAsBytes();
}
callback.failure(originalIndex, region, row, throwable);
}
errors.add(throwable, row, location);
this.hasError.set(true);
}
return canRetry;
}
This will treat the OOM as the canRetry exception.
Thanks!
Best Regards!
jingych
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: CDH5 hbase client outofmemory
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Jingych,
This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details
on you OOME? Did you figured why it occured?
JM
2014-05-22 1:20 GMT-04:00 jingych <ji...@neusoft.com>:
> Hello, everyone!
>
> I found the CDH5 hbase client swallowed the Outofmemory exception.
>
> It didn't throw out, cause the program couldn't make the correct response
> to process the OOM.
>
> Is't good to catch the client OOM? Why?
>
> The CDH5 hbase client process the throwable code:
> private boolean manageError(int originalIndex, Row row, boolean canRetry,
> Throwable throwable, HRegionLocation
> location) {
> if (canRetry && throwable != null && throwable instanceof
> DoNotRetryIOException) {
> canRetry = false;
> }
>
> byte[] region = null;
> if (canRetry && callback != null) {
> region = location == null ? null :
> location.getRegionInfo().getEncodedNameAsBytes();
> canRetry = callback.retriableFailure(originalIndex, row, region,
> throwable);
> }
>
> if (!canRetry) {
> if (callback != null) {
> if (region == null && location != null) {
> region = location.getRegionInfo().getEncodedNameAsBytes();
> }
> callback.failure(originalIndex, region, row, throwable);
> }
> errors.add(throwable, row, location);
> this.hasError.set(true);
> }
>
> return canRetry;
> }
>
> This will treat the OOM as the canRetry exception.
>
> Thanks!
>
> Best Regards!
>
>
>
>
> jingych
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>