You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Erik Rozendaal <dl...@gmail.com> on 2009/10/22 09:33:06 UTC
HBASE-1927 (was Re: HBase 0.20.1 scanners not closing properly (memory leak))
Issue created: HBASE-1927
On 21 okt 2009, at 23:55, stack wrote:
> Thanks for digging in Erik. Nice one. Would you mind making an
> issue of
> your findings? File it against 0.20.2 so we can roll out the fix in
> next
> 0.20.x release.
> St.Ack
>
> On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal
> <dl...@gmail.com>wrote:
>
>> I did some more investigation into this issue since after the
>> original
>> issue stop occuring I noticed that MemStoreScanners where still
>> being leaked
>> when scanning a store with an empty MemStore.
>>
>> The cause looks to be the KeyValueHeap constructor. It drops
>> scanners when
>> the scanner's peek() method returns null (line 58 of KeyValueHeap).
>>
>> Unfortunately some scanners (like StoreScanner and MemStoreScanner)
>> register themselves to some global list when constructed and only
>> deregister
>> on close().
>>
>> So a quick fix may be to add an
>>
>> } else {
>> scanner.close()
>> }
>>
>> to the KeyValueHeap constructor when the scanner is not added. I'm
>> not sure
>> if this is the cleanest fix though...
>>
>> Regards,
>> Erik
>>
>>
>> On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
>>
>> Hello Erik,
>>> I think your attachments were blocked. Could you please upload them
>>> somewhere else?
>>>
>>> On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <dlrozendaal@gmail.com
>>>> wrote:
>>>
>>> Hi all,
>>>>
>>>> After some performance testing on my HBase 0.20.1 development
>>>> environment
>>>> (running in pseudo- and full-distributed mode on a single laptop) I
>>>> noticed
>>>> that scanners do not get closed properly on the region server.
>>>> After
>>>> creating a heap dump with Netbeans I can see the StoreScanner
>>>> instances
>>>> are
>>>> still in the Store.changedReaderObservers collection.
>>>>
>>>> Each StoreScanner instance has the "closed" flag set to false, so
>>>> it
>>>> looks
>>>> like the StoreScanner.close() method was never called.
>>>>
>>>> I double-checked my client code and counted the number of times I
>>>> create
>>>> and close a scanner, and these counts match. I also found this is
>>>> repeatable
>>>> from the hbase shell. Open the shell, scan some table, take a
>>>> heap dump
>>>> and
>>>> you'll find an unclosed StoreScanner objects in the
>>>> Store.changedReaderObservers collection.
>>>>
>>>> I've attached screenshots of the number of StoreScanner instances
>>>> (after
>>>> 30.001 scans) and the Store.changedReaderObservers collection of
>>>> one of
>>>> the
>>>> Stores (notice that the closed field's value is 0 => false).
>>>>
>>>> Ultimately the region server runs out of memory and crashes.
>>>>
>>>> Has anyone experience similar problems?
>>>>
>>>> Regards,
>>>> Erik
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Guilherme
>>>
>>> msn: guigermoglio@hotmail.com
>>> homepage: http://sites.google.com/site/germoglio/
>>>
>>
>>
Re: HBASE-1927 (was Re: HBase 0.20.1 scanners not closing properly
(memory leak))
Posted by Jonathan Gray <jl...@streamy.com>.
Erik,
I just put up a patch with the fix you described and a unit test that
replicates the behavior.
Please test to confirm it works. If so, drop a note in the issue and I
will commit.
Thanks for finding the bug.
JG
Erik Rozendaal wrote:
> Issue created: HBASE-1927
>
> On 21 okt 2009, at 23:55, stack wrote:
>
>> Thanks for digging in Erik. Nice one. Would you mind making an issue of
>> your findings? File it against 0.20.2 so we can roll out the fix in next
>> 0.20.x release.
>> St.Ack
>>
>> On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal
>> <dl...@gmail.com>wrote:
>>
>>> I did some more investigation into this issue since after the original
>>> issue stop occuring I noticed that MemStoreScanners where still being
>>> leaked
>>> when scanning a store with an empty MemStore.
>>>
>>> The cause looks to be the KeyValueHeap constructor. It drops scanners
>>> when
>>> the scanner's peek() method returns null (line 58 of KeyValueHeap).
>>>
>>> Unfortunately some scanners (like StoreScanner and MemStoreScanner)
>>> register themselves to some global list when constructed and only
>>> deregister
>>> on close().
>>>
>>> So a quick fix may be to add an
>>>
>>> } else {
>>> scanner.close()
>>> }
>>>
>>> to the KeyValueHeap constructor when the scanner is not added. I'm
>>> not sure
>>> if this is the cleanest fix though...
>>>
>>> Regards,
>>> Erik
>>>
>>>
>>> On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
>>>
>>> Hello Erik,
>>>> I think your attachments were blocked. Could you please upload them
>>>> somewhere else?
>>>>
>>>> On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <dlrozendaal@gmail.com
>>>>> wrote:
>>>>
>>>> Hi all,
>>>>>
>>>>> After some performance testing on my HBase 0.20.1 development
>>>>> environment
>>>>> (running in pseudo- and full-distributed mode on a single laptop) I
>>>>> noticed
>>>>> that scanners do not get closed properly on the region server. After
>>>>> creating a heap dump with Netbeans I can see the StoreScanner
>>>>> instances
>>>>> are
>>>>> still in the Store.changedReaderObservers collection.
>>>>>
>>>>> Each StoreScanner instance has the "closed" flag set to false, so it
>>>>> looks
>>>>> like the StoreScanner.close() method was never called.
>>>>>
>>>>> I double-checked my client code and counted the number of times I
>>>>> create
>>>>> and close a scanner, and these counts match. I also found this is
>>>>> repeatable
>>>>> from the hbase shell. Open the shell, scan some table, take a heap
>>>>> dump
>>>>> and
>>>>> you'll find an unclosed StoreScanner objects in the
>>>>> Store.changedReaderObservers collection.
>>>>>
>>>>> I've attached screenshots of the number of StoreScanner instances
>>>>> (after
>>>>> 30.001 scans) and the Store.changedReaderObservers collection of
>>>>> one of
>>>>> the
>>>>> Stores (notice that the closed field's value is 0 => false).
>>>>>
>>>>> Ultimately the region server runs out of memory and crashes.
>>>>>
>>>>> Has anyone experience similar problems?
>>>>>
>>>>> Regards,
>>>>> Erik
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Guilherme
>>>>
>>>> msn: guigermoglio@hotmail.com
>>>> homepage: http://sites.google.com/site/germoglio/
>>>>
>>>
>>>
>