RPC Timeout - DoNotRetryIOException

[Siddhartha Jana <si...@gmail.com>]

Hi,

We have a 20-node cluster configured with HBase 0.98.1
While running some basic applications across our large HBase tables, I
observe the DoNotRetryIOException being thrown repeatedly. This is observed
after our Map-Reduce job makes a decent amount of progress. I see some RPC
related bug-fixes mentioned in the release notes of this version and also
see a similar bug-fix note in this
link<https://issues.apache.org/jira/browse/HBASE-10848>.
Could some one please direct us to a solution / alternative HBase version?

I have appended the stack trace below:

Thanks,
Siddhartha Jana
University of Houston


======================
Error: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
OutOfOrderScannerNextException: was there a rpc timeout?
 at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:384)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:221)
 at
org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
 at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
 at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

======================