You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Matt Corgan <mc...@hotpads.com> on 2010/09/30 21:08:38 UTC
deadlock between split and delete
I'm inserting several million rows of a table that has ~50 columns, 31 of
which are days of the month and are often null. For the null ones I issue a
delete instead of a put to make sure any previous data in that column gets
deleted. I'm inserting 1000 rows at a time, so the behavior is to put ~40k
cells, delete ~10k cells, and then flush. I've also tried flushing between
the puts and deletes.
This works fine until the region tries to split, at which point i almost
always get a deadlock. I suspect it's similar to these:
https://issues.apache.org/jira/browse/HBASE-2097
https://issues.apache.org/jira/browse/HBASE-2915
Here are the 2 threads in question from the regionserver:
http://pastebin.com/ddVbHxvP
And the regionserver log (split starts at line 38):
http://pastebin.com/JgvchvYK
This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers are
idle other than this.
If there is actually a deadlock, do you think that waiting x milliseconds
between the puts and the deletes could be a temporary workaround?
Thanks,
Matt
Re: deadlock between split and delete
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yeah that's HBASE-2915, try this release candidate instead
http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
J-D
On Thu, Sep 30, 2010 at 12:08 PM, Matt Corgan <mc...@hotpads.com> wrote:
> I'm inserting several million rows of a table that has ~50 columns, 31 of
> which are days of the month and are often null. For the null ones I issue a
> delete instead of a put to make sure any previous data in that column gets
> deleted. I'm inserting 1000 rows at a time, so the behavior is to put ~40k
> cells, delete ~10k cells, and then flush. I've also tried flushing between
> the puts and deletes.
>
> This works fine until the region tries to split, at which point i almost
> always get a deadlock. I suspect it's similar to these:
>
> https://issues.apache.org/jira/browse/HBASE-2097
> https://issues.apache.org/jira/browse/HBASE-2915
>
> Here are the 2 threads in question from the regionserver:
> http://pastebin.com/ddVbHxvP
>
> And the regionserver log (split starts at line 38):
> http://pastebin.com/JgvchvYK
>
> This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers are
> idle other than this.
>
> If there is actually a deadlock, do you think that waiting x milliseconds
> between the puts and the deletes could be a temporary workaround?
>
> Thanks,
> Matt
>
Re: deadlock between split and delete
Posted by Matt Corgan <mc...@hotpads.com>.
Sorry - that was the lastest release in the downloads section of
hbase.apache.org. I'll upgrade right away.
We're on 1.6.0_17. I've seen warnings about u18 having problems. Do you
recommend u20?
On Thu, Sep 30, 2010 at 4:20 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Indeed, but we never do lock upgrades in HBase.
>
> Lokos like the DR the user was using is just too old. It is time to
> upgrade! Please, please, please don't keep on using older DR... the
> point is to get users who are willing to embark on an upgrade path and
> follow along, otherwise it just creates more support load for us and
> reduces the willingness to do DRs in the future.
>
> -ryan
>
> On Thu, Sep 30, 2010 at 1:00 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > RWLock can indeed deadlock with itself - it doesn't support "lock
> > upgrade" from read to write.
> >
> > -Todd
> >
> > On Thu, Sep 30, 2010 at 12:14 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >> What jvm version are you using? A readwrite lock should not be able to
> >> deadlock on itself like that. There has been known jvm bugs involving
> locks
> >> alas...
> >> On Sep 30, 2010 12:09 PM, "Matt Corgan" <mc...@hotpads.com> wrote:
> >>> I'm inserting several million rows of a table that has ~50 columns, 31
> of
> >>> which are days of the month and are often null. For the null ones I
> issue
> >> a
> >>> delete instead of a put to make sure any previous data in that column
> gets
> >>> deleted. I'm inserting 1000 rows at a time, so the behavior is to put
> ~40k
> >>> cells, delete ~10k cells, and then flush. I've also tried flushing
> between
> >>> the puts and deletes.
> >>>
> >>> This works fine until the region tries to split, at which point i
> almost
> >>> always get a deadlock. I suspect it's similar to these:
> >>>
> >>> https://issues.apache.org/jira/browse/HBASE-2097
> >>> https://issues.apache.org/jira/browse/HBASE-2915
> >>>
> >>> Here are the 2 threads in question from the regionserver:
> >>> http://pastebin.com/ddVbHxvP
> >>>
> >>> And the regionserver log (split starts at line 38):
> >>> http://pastebin.com/JgvchvYK
> >>>
> >>> This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers
> are
> >>> idle other than this.
> >>>
> >>> If there is actually a deadlock, do you think that waiting x
> milliseconds
> >>> between the puts and the deletes could be a temporary workaround?
> >>>
> >>> Thanks,
> >>> Matt
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
Re: deadlock between split and delete
Posted by Ryan Rawson <ry...@gmail.com>.
Indeed, but we never do lock upgrades in HBase.
Lokos like the DR the user was using is just too old. It is time to
upgrade! Please, please, please don't keep on using older DR... the
point is to get users who are willing to embark on an upgrade path and
follow along, otherwise it just creates more support load for us and
reduces the willingness to do DRs in the future.
-ryan
On Thu, Sep 30, 2010 at 1:00 PM, Todd Lipcon <to...@cloudera.com> wrote:
> RWLock can indeed deadlock with itself - it doesn't support "lock
> upgrade" from read to write.
>
> -Todd
>
> On Thu, Sep 30, 2010 at 12:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
>> What jvm version are you using? A readwrite lock should not be able to
>> deadlock on itself like that. There has been known jvm bugs involving locks
>> alas...
>> On Sep 30, 2010 12:09 PM, "Matt Corgan" <mc...@hotpads.com> wrote:
>>> I'm inserting several million rows of a table that has ~50 columns, 31 of
>>> which are days of the month and are often null. For the null ones I issue
>> a
>>> delete instead of a put to make sure any previous data in that column gets
>>> deleted. I'm inserting 1000 rows at a time, so the behavior is to put ~40k
>>> cells, delete ~10k cells, and then flush. I've also tried flushing between
>>> the puts and deletes.
>>>
>>> This works fine until the region tries to split, at which point i almost
>>> always get a deadlock. I suspect it's similar to these:
>>>
>>> https://issues.apache.org/jira/browse/HBASE-2097
>>> https://issues.apache.org/jira/browse/HBASE-2915
>>>
>>> Here are the 2 threads in question from the regionserver:
>>> http://pastebin.com/ddVbHxvP
>>>
>>> And the regionserver log (split starts at line 38):
>>> http://pastebin.com/JgvchvYK
>>>
>>> This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers are
>>> idle other than this.
>>>
>>> If there is actually a deadlock, do you think that waiting x milliseconds
>>> between the puts and the deletes could be a temporary workaround?
>>>
>>> Thanks,
>>> Matt
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: deadlock between split and delete
Posted by Todd Lipcon <to...@cloudera.com>.
RWLock can indeed deadlock with itself - it doesn't support "lock
upgrade" from read to write.
-Todd
On Thu, Sep 30, 2010 at 12:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
> What jvm version are you using? A readwrite lock should not be able to
> deadlock on itself like that. There has been known jvm bugs involving locks
> alas...
> On Sep 30, 2010 12:09 PM, "Matt Corgan" <mc...@hotpads.com> wrote:
>> I'm inserting several million rows of a table that has ~50 columns, 31 of
>> which are days of the month and are often null. For the null ones I issue
> a
>> delete instead of a put to make sure any previous data in that column gets
>> deleted. I'm inserting 1000 rows at a time, so the behavior is to put ~40k
>> cells, delete ~10k cells, and then flush. I've also tried flushing between
>> the puts and deletes.
>>
>> This works fine until the region tries to split, at which point i almost
>> always get a deadlock. I suspect it's similar to these:
>>
>> https://issues.apache.org/jira/browse/HBASE-2097
>> https://issues.apache.org/jira/browse/HBASE-2915
>>
>> Here are the 2 threads in question from the regionserver:
>> http://pastebin.com/ddVbHxvP
>>
>> And the regionserver log (split starts at line 38):
>> http://pastebin.com/JgvchvYK
>>
>> This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers are
>> idle other than this.
>>
>> If there is actually a deadlock, do you think that waiting x milliseconds
>> between the puts and the deletes could be a temporary workaround?
>>
>> Thanks,
>> Matt
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: deadlock between split and delete
Posted by Ryan Rawson <ry...@gmail.com>.
What jvm version are you using? A readwrite lock should not be able to
deadlock on itself like that. There has been known jvm bugs involving locks
alas...
On Sep 30, 2010 12:09 PM, "Matt Corgan" <mc...@hotpads.com> wrote:
> I'm inserting several million rows of a table that has ~50 columns, 31 of
> which are days of the month and are often null. For the null ones I issue
a
> delete instead of a put to make sure any previous data in that column gets
> deleted. I'm inserting 1000 rows at a time, so the behavior is to put ~40k
> cells, delete ~10k cells, and then flush. I've also tried flushing between
> the puts and deletes.
>
> This works fine until the region tries to split, at which point i almost
> always get a deadlock. I suspect it's similar to these:
>
> https://issues.apache.org/jira/browse/HBASE-2097
> https://issues.apache.org/jira/browse/HBASE-2915
>
> Here are the 2 threads in question from the regionserver:
> http://pastebin.com/ddVbHxvP
>
> And the regionserver log (split starts at line 38):
> http://pastebin.com/JgvchvYK
>
> This happens in both 0.20.6 and 0.89.20100726. 5 GB heap, and servers are
> idle other than this.
>
> If there is actually a deadlock, do you think that waiting x milliseconds
> between the puts and the deletes could be a temporary workaround?
>
> Thanks,
> Matt