You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Adam Silberstein <si...@yahoo-inc.com> on 2009/12/18 06:36:22 UTC

Scan api result loop problem

Hi,
I wrote some simple client code to parse scan results, and it seems to be
causing heavy CPU usage on my machine.  I¹ve commented out most of my code,
and have this left:

ResultScanner scanner = null;
//some code to set scanner

for (Result rr: scanner) {

}
scanner.close();

If I comment out the loop, then the CPU problems go away.  If I keep it, but
have nothing inside, I see the problem.

I saw a bug mentioned in release notes that talk about a memory leak in scan
( 
http://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&sty
leName=Html&version=12314307).  But it appears that it was fixed in 0.20.2,
which is the verison I am using.

Has anyone else noticed this, and have any suggestions?

Thanks,
Adam

Re: Scan api result loop problem

Posted by stack <st...@duboce.net>.
On Sat, Dec 19, 2009 at 8:07 PM, Adam Silberstein <si...@yahoo-inc.com>wrote:

> ...
> -My bail out code should keep me from going to the bitter end.


Yes.



>  I tried
> bailing out at different numbers (2, 100).  I'm seeing way better
> throughput.


Yeah.  You want to balance rpc calls and over-fetching.



> CPU still strikes me as high, but maybe I'm underestimating
> what's going on under the covers.


We're probably heavy-weight here.  We're doing a bit of fancy footwork so we
ride the scanner across regions and so we do right thing regards caching.



> I have an 8-core machine.  With 10
> parallel clients doing 100 record scans, usage across each core adds up to
> 60-70%   Does that seem reasonable?
>

I'm not sure.  Can you hang a profiler on it to see where the CPU is being
chewed up?

Result is backed by a raw list of KeyValues.  You have access to it even.
 I'd guess that working w/ the raw list, you'd save CPU but not sure you can
do that via Scanner#next (maybe someone else knows better).

Go easy,
St.Ack

Re: Scan api result loop problem

Posted by Adam Silberstein <si...@yahoo-inc.com>.
Hi,
Interesting! I think you're right, thanks.

I forgot to mention that I'm also using setCaching() function to control
number of records that come back with one call.  And I'm typically setting
that to somewhere between 1 and 1000.  I just did another test where I set
caching to 2 every time.  Also, in my loop I added code that bails out of
the top of the loop if I went through it 2 previous times.

Here's what I observe:
-My bail out code gets triggered.  So I take this to mean that although I
did setCaching(2), the client makes repeated calls to get more results.  So
I probably was getting scan results to the bitter end, as you diagnosed.
-My bail out code should keep me from going to the bitter end.  I tried
bailing out at different numbers (2, 100).  I'm seeing way better
throughput.  CPU still strikes me as high, but maybe I'm underestimating
what's going on under the covers.  I have an 8-core machine.  With 10
parallel clients doing 100 record scans, usage across each core adds up to
60-70%   Does that seem reasonable?

Thanks,
Adam


On 12/19/09 8:42 AM, "stack" <st...@duboce.net> wrote:

> On Thu, Dec 17, 2009 at 9:36 PM, Adam Silberstein
> <si...@yahoo-inc.com>wrote:
> 
>> Hi,
>> I wrote some simple client code to parse scan results, and it seems to be
>> causing heavy CPU usage on my machine.  I¹ve commented out most of my code,
>> and have this left:
>> 
>> ResultScanner scanner = null;
>> //some code to set scanner
>> 
>> for (Result rr: scanner) {
>> 
>> }
>> scanner.close();
>> 
>> 
> The above will run through the scanner nexting composing Result objects --
> see ClientScanner#next -- all the way down to its bitter end.  Is this whats
> burning your CPU?
> 
> St.Ack
> 
> 
> 
> 
> 
>> If I comment out the loop, then the CPU problems go away.  If I keep it,
>> but
>> have nothing inside, I see the problem.
>> 
>> I saw a bug mentioned in release notes that talk about a memory leak in
>> scan
>> (
>> 
>> http://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&sty
>> leName=Html&version=12314307).  But it appears that it was fixed in
>> 0.20.2,
>> which is the verison I am using.
>> 
>> Has anyone else noticed this, and have any suggestions?
>> 
>> Thanks,
>> Adam
>> 


Re: Scan api result loop problem

Posted by stack <st...@duboce.net>.
On Thu, Dec 17, 2009 at 9:36 PM, Adam Silberstein <si...@yahoo-inc.com>wrote:

> Hi,
> I wrote some simple client code to parse scan results, and it seems to be
> causing heavy CPU usage on my machine.  I¹ve commented out most of my code,
> and have this left:
>
> ResultScanner scanner = null;
> //some code to set scanner
>
> for (Result rr: scanner) {
>
> }
> scanner.close();
>
>
The above will run through the scanner nexting composing Result objects --
see ClientScanner#next -- all the way down to its bitter end.  Is this whats
burning your CPU?

St.Ack





> If I comment out the loop, then the CPU problems go away.  If I keep it,
> but
> have nothing inside, I see the problem.
>
> I saw a bug mentioned in release notes that talk about a memory leak in
> scan
> (
>
> http://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&sty
> leName=Html&version=12314307).  But it appears that it was fixed in
> 0.20.2,
> which is the verison I am using.
>
> Has anyone else noticed this, and have any suggestions?
>
> Thanks,
> Adam
>