You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/12/04 19:46:44 UTC

[jira] Created: (HBASE-1045) Hangup by regionserver causes write to fail

Hangup by regionserver causes write to fail
-------------------------------------------

                 Key: HBASE-1045
                 URL: https://issues.apache.org/jira/browse/HBASE-1045
             Project: Hadoop HBase
          Issue Type: Bug
          Components: client
            Reporter: Andrew Purtell
             Fix For: 0.19.0


Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?

Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
	at org.archive.modules.Processor.process(Processor.java:123)
	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
	at org.apache.hadoop.ipc.Client.call(Client.java:699)
	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
	at $Proxy12.batchUpdates(Unknown Source)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
	at org.apache.hadoop.ipc.Client.call(Client.java:685)
	... 16 more


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653434#action_12653434 ] 

Andrew Purtell commented on HBASE-1045:
---------------------------------------

It does not look like the client retried, otherwise the top level exception would be a RetriesExceededException, correct?



> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658429#action_12658429 ] 

Jean-Daniel Cryans commented on HBASE-1045:
-------------------------------------------

Mmm, that seems very strange. Line 911 is :

{code}
int index = getRegionServerForWithoutRetries(new ServerCallable<Integer>(
              this, tableName, batchUpdate.getRow()) {
{code}

Which would mean that the batchUpdate was null, but we already do that check in HTable! And it would have failed some lines before.

Andrew, I will need more info like if it was using the batching feature for updates, and also I would need that trace with DEBUG if possible.
I tried recreating the problem here with PE and auto-commit enabled but it passed (sadly).

Thx.

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans updated HBASE-1045:
--------------------------------------

    Attachment: hbase-1045-1.patch

Patch adds the retry of only 1 row when IOE or NSRE and makes the getRegionServerForWithoutRetries eat the IOE instead of throwing it like it did for Andrew.

Here is what is now happening when a region splits during PE with some custom debug:

{code}
...
08/12/20 13:42:39 DEBUG client.HConnectionManager$TableServers: Commit 10478 rows starting at 0000513422
08/12/20 13:42:47 DEBUG client.HConnectionManager$TableServers: Index returned is 1946
08/12/20 13:42:47 DEBUG client.HConnectionManager$TableServers: Reloading table servers because region server didn't accept updates; tries=0 of max=10, waiting=2000ms
08/12/20 13:42:49 DEBUG client.HConnectionManager$TableServers: Removed TestTable,,1229798498529 from cache because of 0000515368
08/12/20 13:42:49 DEBUG client.HConnectionManager$TableServers: Commit 1 rows starting at 0000515368
08/12/20 13:42:49 DEBUG client.HConnectionManager$TableServers: Index returned is 0
08/12/20 13:42:49 DEBUG client.HConnectionManager$TableServers: Reloading table servers because region server didn't accept updates; tries=1 of max=10, waiting=2000ms
08/12/20 13:42:51 DEBUG client.HConnectionManager$TableServers: Removed TestTable,,1229798498529 from cache because of 0000515368
08/12/20 13:42:51 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region TestTable,0000085888,1229798566468>. Retrying after sleep of 2000
08/12/20 13:42:51 DEBUG client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0000515368,99999999999999
08/12/20 13:42:51 DEBUG client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}
08/12/20 13:42:53 DEBUG client.HConnectionManager$TableServers: Commit 1 rows starting at 0000515368
08/12/20 13:42:53 DEBUG client.HConnectionManager$TableServers: Index returned is -1
08/12/20 13:42:53 DEBUG client.HConnectionManager$TableServers: Commit 8531 rows starting at 0000515369
08/12/20 13:42:53 DEBUG client.HConnectionManager$TableServers: Index returned is -1
...
{code}

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658205#action_12658205 ] 

stack commented on HBASE-1045:
------------------------------

(Following on from chat w/ J-D up on IRC)

I do not see the infinite retries J-D.  I see this:

{code}
          final BatchUpdate[] updates = tempUpdates.toArray(new BatchUpdate[0]);
          int index = getRegionServerForWithoutRetries(new ServerCallable<Integer>(
              this, tableName, batchUpdate.getRow()) {
            public Integer call() throws IOException {
              int i = server.batchUpdates(location.getRegionInfo()
                  .getRegionName(), updates);
              return i;
            }
          });
{code}

... around line #912 of HCM.  If exception thrown, it comes all the ways out.

Could we do your #3 above and change getRegionServerForWithoutRetries somehow to something that retries -- IOEs included -- but that if it fails retrying, that it fails on the batch; it does not move on to the next BatchUpdate?

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658434#action_12658434 ] 

Andrew Purtell commented on HBASE-1045:
---------------------------------------

Sorry, I'm running the 0.19 on 0.18 branch. Can't run plain trunk on my cluster. Line 911 is:

        if (!isLastRow) {
            region = getRegionLocation(tableName, list.get(i+1).getRow, false)
                .getRegionInfo().getRegionName();
        }
>>> if (!Bytes.equals(currentRegion, region) || isLastRow) {
            final BatchUpdate[] updates = tempUpdates.toArray(new BatchUpdate[0]);

At the time some regions of the table were offline. I had to manually null out info:server in META to get them open. 

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658307#action_12658307 ] 

stack commented on HBASE-1045:
------------------------------

Are you going to add retry of IOEs J-D?

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653416#action_12653416 ] 

stack commented on HBASE-1045:
------------------------------

Did the client not retry?  If not, it should treat these errors as any other.

Will always be case where client fails even after N retries but thats a given here, right?  Or would you suggest client should just try forever?

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658305#action_12658305 ] 

Jean-Daniel Cryans commented on HBASE-1045:
-------------------------------------------

Oh for some weird reason I was looking at an old version of flushCommits so #1 is already implemented in HCM.processBatchOfRows

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653581#action_12653581 ] 

Andrew Purtell commented on HBASE-1045:
---------------------------------------

Had a hbase-writer go back to the same zombie regionserver 229 times. All 229 writes failed. 

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans resolved HBASE-1045.
---------------------------------------

    Resolution: Fixed
      Assignee: Jean-Daniel Cryans

Committed to trunk.

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653581#action_12653581 ] 

apurtell edited comment on HBASE-1045 at 12/4/08 5:30 PM:
----------------------------------------------------------------

Had a hbase-writer go back to the same zombie regionserver 229 times. All 229 writes failed. Another one for 418, another for 122, another for 367.

      was (Author: apurtell):
    Had a hbase-writer go back to the same zombie regionserver 229 times. All 229 writes failed. 
  
> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658330#action_12658330 ] 

stack commented on HBASE-1045:
------------------------------

Looks good to me J-D.

Andrew, to answer your original question, yes, its possible for client to fail.   W/ J-Ds patch, we'll retry all types of exceptions but will give up if we continue to fail after N attempts.

Is the J-D patch enough to close this issue?

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653587#action_12653587 ] 

Andrew Purtell commented on HBASE-1045:
---------------------------------------

Based on the regionserver log in question it looks like the shutdown thread completed quickly after OOME (in server select in IPC). So for some reason the region was not reassigned in a timely manner. Clients kept finding the old region information in meta and went to the server but there was no listener on the socket, hence:

SEVERE: Failed write of Record .... java.io.IOException: Call to 10.30.94.37:60020 failed on local exception: Connection refused. 

I did not have our service recovery framework running. Otherwise another region server would have been launched and would have thrown NSREs (eventually). 


> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658225#action_12658225 ] 

Jean-Daniel Cryans commented on HBASE-1045:
-------------------------------------------

Stack,

Maybe I wasn't clear in what I said but the IOE are thrown all the way, it's the NSRE that's retried infinitely.

bq. Could we do your #3 above and change getRegionServerForWithoutRetries somehow to something that retries - IOEs included - but that if it fails retrying, that it fails on the batch; it does not move on to the next BatchUpdate?

Thinking about it, it's the fact the I wanted to retry the whole bunch of rows that had me doing my little hack. Retrying only 1 row will be slower in certain occasions but less prone to OOME's.

I will work on that this w/e.

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658198#action_12658198 ] 

Jean-Daniel Cryans commented on HBASE-1045:
-------------------------------------------

Currently the way the retries are handled with the batched updates is that it will get retried infinitely with 1 second between each call to a HRS. This hack is there until we can refactor the handling of getting/updating batches of rows, What happened to Andrew is that he got an IOE after 100 and 200+ NSRE which are always retried until an IOE was thrown which, currently, is not catch.

What we could do to get the situation better :

- Add a max number of retries and use the exponential backoff. This would worsen the hack by duplicating the HCM stuff in HTable but we will refactor it for 0.20.

- Catch the IOE now that it won't get retried forever.

- If we are committing a batch of rows, we should do the retries with only 1 row so that we don't OOME region servers.

This way, a single row commit would at least know what failed. 

And a nice to have to be to inform HMaster that something is wrong with a HRS.

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658397#action_12658397 ] 

Andrew Purtell commented on HBASE-1045:
---------------------------------------

+1

Thanks!


> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658443#action_12658443 ] 

Jean-Daniel Cryans commented on HBASE-1045:
-------------------------------------------

So in your case the region returned by getRegionLocation would have been null, am I reading this right? Is it even possible?

Also do you want me to port my patch to your branch?

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell resolved HBASE-1045.
-----------------------------------

    Resolution: Fixed

NPE addressed elsewhere. No need to port. Thanks J-D.

> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Reopened: (HBASE-1045) Hangup by regionserver causes write to fail

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell reopened HBASE-1045:
-----------------------------------


Dec 21, 2008 7:28:07 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
SEVERE: Failed write of Record: dns:www.mp3.com (in thread 'ToeThread #3: dns:www.mp3.com'; in processor 'Archiver')
java.io.IOException: java.lang.NullPointerException
        at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
        at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
        at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
        at org.archive.modules.Processor.process(Processor.java:123)
        at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:911)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
        at net.iridiant.content.Content.store(Unknown Source)
        at net.iridiant.content.Content.store(Unknown Source)
        ... 6 more


> Hangup by regionserver causes write to fail
> -------------------------------------------
>
>                 Key: HBASE-1045
>                 URL: https://issues.apache.org/jira/browse/HBASE-1045
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>            Reporter: Andrew Purtell
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.19.0
>
>         Attachments: hbase-1045-1.patch
>
>
> Root cause is OOME on the region server. Nonetheless a hangup during IPC causes the client to fail the write, currently causing data loss. Should the application catch and retry? Or should the client libraries try harder?
> Dec 4, 2008 5:25:30 PM com.powerset.heritrix.writer.HBaseWriterProcessor innerProcessResult
> SEVERE: Failed write of Record: http://www.publicrecordslocal.com/georgia.htm (in thread 'ToeThread #9: http://www.publicrecordslocal.com/georgia.htm'; in processor 'Archiver')
> java.io.IOException: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at com.powerset.heritrix.writer.HBaseWriter.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.write(Unknown Source)
> 	at com.powerset.heritrix.writer.HBaseWriterProcessor.innerProcessResult(Unknown Source)
> 	at org.archive.modules.Processor.process(Processor.java:123)
> 	at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:310)
> 	at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)
> Caused by: java.io.IOException: Call to /10.30.94.38:60020 failed on local exception: Connection refused
> 	at org.apache.hadoop.ipc.Client.call(Client.java:699)
> 	at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:323)
> 	at $Proxy12.batchUpdates(Unknown Source)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:919)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.call(HConnectionManager.java:917)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerForWithoutRetries(HConnectionManager.java:875)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:916)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1267)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1238)
> 	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1218)
> 	at net.iridiant.content.Content.storeURLInfo(Unknown Source)
> 	... 6 more
> Caused by: java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:685)
> 	... 16 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.