You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/10/16 03:32:31 UTC

[jira] Created: (HBASE-1910) IndexedRegion RPC deadlock

IndexedRegion RPC deadlock
--------------------------

                 Key: HBASE-1910
                 URL: https://issues.apache.org/jira/browse/HBASE-1910
             Project: Hadoop HBase
          Issue Type: Bug
    Affects Versions: 0.20.1, 0.21.0
            Reporter: Andrew Purtell


>From Tatsuya Kawano up on hbase-user@

{quote}
50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded.

I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex().

I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having.

{code}
  // FIXME: This call takes place in an RPC, and requires an RPC. This makes for
  // a likely deadlock if the number of RPCs we are trying to serve is >= the
  // number of handler threads.
  private void updateIndex(IndexSpecification indexSpec, byte[] row,
      SortedMap<byte[], byte[]> columnValues) throws IOException {
{code}

I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default).

Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml)
{quote}

Opening this issue to track the FIXME.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1910) IndexedRegion RPC deadlock

Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766368#action_12766368 ] 

ryan rawson commented on HBASE-1910:
------------------------------------

I doubt we could fix this in 0.20.x, (especially not in 0.20.1), since it would require a completely different RPC architecture....

I think this might be fixable in 0.22.

> IndexedRegion RPC deadlock
> --------------------------
>
>                 Key: HBASE-1910
>                 URL: https://issues.apache.org/jira/browse/HBASE-1910
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.21.0
>            Reporter: Andrew Purtell
>
> From Tatsuya Kawano up on hbase-user@
> {quote}
> 50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded.
> I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex().
> I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having.
> {code}
>   // FIXME: This call takes place in an RPC, and requires an RPC. This makes for
>   // a likely deadlock if the number of RPCs we are trying to serve is >= the
>   // number of handler threads.
>   private void updateIndex(IndexSpecification indexSpec, byte[] row,
>       SortedMap<byte[], byte[]> columnValues) throws IOException {
> {code}
> I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default).
> Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml)
> {quote}
> Opening this issue to track the FIXME.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1910) IndexedRegion RPC deadlock

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-1910:
----------------------------------

    Attachment: thread.dump.gz

Attached thread dump provided by user for reference.

> IndexedRegion RPC deadlock
> --------------------------
>
>                 Key: HBASE-1910
>                 URL: https://issues.apache.org/jira/browse/HBASE-1910
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.21.0
>            Reporter: Andrew Purtell
>             Fix For: 0.22.0
>
>         Attachments: thread.dump.gz
>
>
> From Tatsuya Kawano up on hbase-user@
> {quote}
> 50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded.
> I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex().
> I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having.
> {code}
>   // FIXME: This call takes place in an RPC, and requires an RPC. This makes for
>   // a likely deadlock if the number of RPCs we are trying to serve is >= the
>   // number of handler threads.
>   private void updateIndex(IndexSpecification indexSpec, byte[] row,
>       SortedMap<byte[], byte[]> columnValues) throws IOException {
> {code}
> I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default).
> Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml)
> {quote}
> Opening this issue to track the FIXME.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1910) IndexedRegion RPC deadlock

Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ryan rawson updated HBASE-1910:
-------------------------------

    Fix Version/s: 0.22.0

> IndexedRegion RPC deadlock
> --------------------------
>
>                 Key: HBASE-1910
>                 URL: https://issues.apache.org/jira/browse/HBASE-1910
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.21.0
>            Reporter: Andrew Purtell
>             Fix For: 0.22.0
>
>
> From Tatsuya Kawano up on hbase-user@
> {quote}
> 50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded.
> I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex().
> I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having.
> {code}
>   // FIXME: This call takes place in an RPC, and requires an RPC. This makes for
>   // a likely deadlock if the number of RPCs we are trying to serve is >= the
>   // number of handler threads.
>   private void updateIndex(IndexSpecification indexSpec, byte[] row,
>       SortedMap<byte[], byte[]> columnValues) throws IOException {
> {code}
> I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default).
> Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml)
> {quote}
> Opening this issue to track the FIXME.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.