Exceptions were thrown when I Query a big HBase table in Impala

[Tao Xiao <xi...@gmail.com>]

I'm using CDH 5.0.2. I created an external table named "impala_AA" in Hive
shell and mapped it to a HBase table named "AA".

When AA had 10,000 rows (each row had 30 columns), I can query records in
AA from Impala shell, for example, select * from impala_AA where
col_9='3687'.

But when AA had more records (say, 100,000 rows), when I tried to query
using the same SQL statements in Impala shell, exceptions where thrown as
follows:

Query: select * from impala_aa  where col_9='3687'
ERRORS: DoNotRetryIOException: Failed after retry of
OutOfOrderScannerNextException: was there a rpc timeout?
CAUSED BY: OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
909966436422316992 number_of_rows: 1024 close_scanner: false next_call_seq:
0
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3018)
 at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
 at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
CAUSED BY: RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
909966436422316992 number_of_rows: 1024 close_scanner: false next_call_seq:
0
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3018)
 at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
 at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)

Backend 0:DoNotRetryIOException: Failed after retry of
OutOfOrderScannerNextException: was there a rpc timeout?
CAUSED BY: OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
909966436422316992 number_of_rows: 1024 close_scanner: false next_call_seq:
0
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3018)
 at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
 at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
CAUSED BY: RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
909966436422316992 number_of_rows: 1024 close_scanner: false next_call_seq:
0
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3018)
 at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
 at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)


Was that because HBase table AA had too many records?

In the exception thrown above, we can see "OutOfOrderScannerNextException",
according to this page
<http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/exceptions/OutOfOrderScannerNextException.html>
, OutOfOrderScannerNextException is thrown by a RegionServer while doing
next() calls on a ResultScanner. Both client and server maintain a
nextCallSeq and if they do not match, RS will throw this exception.

I'm not sure why this exception would be thrown each time I query a
relatively big HBase table using Impala query statement.