scan + filter failed with OutOfOrderScannerNextException

[Henry Hung <YT...@winbond.com>]

Hi All,

Is there a way to let scanner finish scanning all regions without throwing this kind of error? I'm using scan with filter MUST_PASS_ALL, and I observe that whenever the result data is smaller (let's say 10%) compare to another filter with larger result (let's say 80%), it always failed.

org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:391)
at com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(MesPerfLogTimeResultSet.java:58)
at com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(MesPerfLogTimeResultSet.java:1)
at com.winbond.hbase.executor.Program.main(Program.java:106)
Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq: 0
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3007)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:235)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:343)
... 3 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException): org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq: 0
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3007)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)

at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:27332)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:167)
... 7 more

Best regards,
Henry

________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.

Re: scan + filter failed with OutOfOrderScannerNextException

[Anoop John <an...@gmail.com>]

Just after the 1st instance of such an exception at client side, ur scan
failed?  Because on receiving this Exception, we retry with a new Scanner
automatically.  I am not sure whether all of ur retry also fails.

I think it will retry with new Scanner only one more time. (remembering
some work like this done) So might be like that also failed.  So pls try
upping the client time out

-Anoop-

On Mon, Sep 29, 2014 at 3:25 PM, Qiang Tian <ti...@gmail.com> wrote:

> How many threads in your client?
>
> On Mon, Sep 29, 2014 at 4:05 PM, Henry Hung <YT...@winbond.com> wrote:
>
> > Hi Anoop,
> >
> > I receive this error in client side, and pretty sure the scan failed.
> > I'm using default caching, so it should be 100, right?
> > About scan time out period, I will try to set it higher, probably 1 hour.
> >
> > BTW, I'm using hbase 0.96.0.
> >
> > Best regards,
> > Henry
> >
> > -----Original Message-----
> > From: Anoop John [mailto:anoop.hbase@gmail.com]
> > Sent: Monday, September 29, 2014 3:13 PM
> > To: user@hbase.apache.org
> > Subject: Re: scan + filter failed with OutOfOrderScannerNextException
> >
> > Hi
> >    Even when the RS throws this Exception, the client side will start a
> > new Scanner and retry. U just see this in log or the scan is failing
> > altogether?
> > What is the caching you use on Scan?  When most of the rows are filtered
> > out at server side, it takes more time to fetch and return the 'caching'
> > number of rows back to client. By this time the client would have already
> > timed out and sent a retry. We allow such a retry will cause not getting
> > back all the results. This is a serious problem and this Exception way
> > helps there.   You can try increasing the scan time out period at client
> > side. This should help.
> >
> > -Anoop-
> >
> > On Mon, Sep 29, 2014 at 12:35 PM, Henry Hung <YT...@winbond.com>
> wrote:
> >
> > > Hi All,
> > >
> > > Is there a way to let scanner finish scanning all regions without
> > > throwing this kind of error? I'm using scan with filter MUST_PASS_ALL,
> > > and I observe that whenever the result data is smaller (let's say 10%)
> > > compare to another filter with larger result (let's say 80%), it always
> > failed.
> > >
> > > org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> > > OutOfOrderScannerNextException: was there a rpc timeout?
> > > at
> > > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> > > 91)
> > > at
> > > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> > > esPerfLogTimeResultSet.java:58)
> > > at
> > > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> > > esPerfLogTimeResultSet.java:1) at
> > > com.winbond.hbase.executor.Program.main(Program.java:106)
> > > Caused by:
> > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > > Expected
> > > nextCallSeq: 1 But the nextCallSeq got from client: 0;
> > request=scanner_id:
> > > 1594996183743944787 number_of_rows: 100 close_scanner: false
> > next_call_seq:
> > > 0
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> > > java:3007)
> > > at
> > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > > 2.callBlockingMethod(ClientProtos.java:26929)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> > >
> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method) at
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> > > rAccessorImpl.java:39)
> > > at
> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> > > nstructorAccessorImpl.java:27) at
> > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at
> > > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> > > tion.java:106)
> > > at
> > > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> > > ption.java:95)
> > > at
> > > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(Proto
> > > bufUtil.java:235)
> > > at
> > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > > va:197)
> > > at
> > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > > va:57)
> > > at
> > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> > > tryingCaller.java:116)
> > > at
> > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> > > tryingCaller.java:94)
> > > at
> > > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> > > 43)
> > > ... 3 more
> > > Caused by:
> > >
> >
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
> > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > > Expected
> > > nextCallSeq: 1 But the nextCallSeq got from client: 0;
> > request=scanner_id:
> > > 1594996183743944787 number_of_rows: 100 close_scanner: false
> > next_call_seq:
> > > 0
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> > > java:3007)
> > > at
> > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > > 2.callBlockingMethod(ClientProtos.java:26929)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.jav
> > > a:1650)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation
> > > .callBlockingMethod(RpcClient.java:1708)
> > > at
> > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > > BlockingStub.scan(ClientProtos.java:27332)
> > > at
> > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > > va:167)
> > > ... 7 more
> > >
> > > Best regards,
> > > Henry
> > >
> > > ________________________________
> > > The privileged confidential information contained in this email is
> > > intended for use only by the addressees as indicated by the original
> > > sender of this email. If you are not the addressee indicated in this
> > > email or are not responsible for delivery of the email to such a
> > > person, please kindly reply to the sender indicating this fact and
> > > delete all copies of it from your computer and network server
> > > immediately. Your cooperation is highly appreciated. It is advised
> > > that any unauthorized use of confidential information of Winbond is
> > > strictly prohibited; and any information in this email irrelevant to
> > > the official business of Winbond shall be deemed as neither given nor
> > endorsed by Winbond.
> > >
> >
> > The privileged confidential information contained in this email is
> > intended for use only by the addressees as indicated by the original
> sender
> > of this email. If you are not the addressee indicated in this email or
> are
> > not responsible for delivery of the email to such a person, please kindly
> > reply to the sender indicating this fact and delete all copies of it from
> > your computer and network server immediately. Your cooperation is highly
> > appreciated. It is advised that any unauthorized use of confidential
> > information of Winbond is strictly prohibited; and any information in
> this
> > email irrelevant to the official business of Winbond shall be deemed as
> > neither given nor endorsed by Winbond.
> >
>

Re: scan + filter failed with OutOfOrderScannerNextException

[Qiang Tian <ti...@gmail.com>]

How many threads in your client?

On Mon, Sep 29, 2014 at 4:05 PM, Henry Hung <YT...@winbond.com> wrote:

> Hi Anoop,
>
> I receive this error in client side, and pretty sure the scan failed.
> I'm using default caching, so it should be 100, right?
> About scan time out period, I will try to set it higher, probably 1 hour.
>
> BTW, I'm using hbase 0.96.0.
>
> Best regards,
> Henry
>
> -----Original Message-----
> From: Anoop John [mailto:anoop.hbase@gmail.com]
> Sent: Monday, September 29, 2014 3:13 PM
> To: user@hbase.apache.org
> Subject: Re: scan + filter failed with OutOfOrderScannerNextException
>
> Hi
>    Even when the RS throws this Exception, the client side will start a
> new Scanner and retry. U just see this in log or the scan is failing
> altogether?
> What is the caching you use on Scan?  When most of the rows are filtered
> out at server side, it takes more time to fetch and return the 'caching'
> number of rows back to client. By this time the client would have already
> timed out and sent a retry. We allow such a retry will cause not getting
> back all the results. This is a serious problem and this Exception way
> helps there.   You can try increasing the scan time out period at client
> side. This should help.
>
> -Anoop-
>
> On Mon, Sep 29, 2014 at 12:35 PM, Henry Hung <YT...@winbond.com> wrote:
>
> > Hi All,
> >
> > Is there a way to let scanner finish scanning all regions without
> > throwing this kind of error? I'm using scan with filter MUST_PASS_ALL,
> > and I observe that whenever the result data is smaller (let's say 10%)
> > compare to another filter with larger result (let's say 80%), it always
> failed.
> >
> > org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> > OutOfOrderScannerNextException: was there a rpc timeout?
> > at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> > 91)
> > at
> > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> > esPerfLogTimeResultSet.java:58)
> > at
> > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> > esPerfLogTimeResultSet.java:1) at
> > com.winbond.hbase.executor.Program.main(Program.java:106)
> > Caused by:
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > Expected
> > nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id:
> > 1594996183743944787 number_of_rows: 100 close_scanner: false
> next_call_seq:
> > 0
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> > java:3007)
> > at
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > 2.callBlockingMethod(ClientProtos.java:26929)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> > at
> > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> >
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method) at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> > rAccessorImpl.java:39)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> > nstructorAccessorImpl.java:27) at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> > tion.java:106)
> > at
> > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> > ption.java:95)
> > at
> > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(Proto
> > bufUtil.java:235)
> > at
> > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > va:197)
> > at
> > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > va:57)
> > at
> > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> > tryingCaller.java:116)
> > at
> > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> > tryingCaller.java:94)
> > at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> > 43)
> > ... 3 more
> > Caused by:
> >
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > Expected
> > nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id:
> > 1594996183743944787 number_of_rows: 100 close_scanner: false
> next_call_seq:
> > 0
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> > java:3007)
> > at
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > 2.callBlockingMethod(ClientProtos.java:26929)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> > at
> > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> >
> > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
> > at
> > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.jav
> > a:1650)
> > at
> > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation
> > .callBlockingMethod(RpcClient.java:1708)
> > at
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > BlockingStub.scan(ClientProtos.java:27332)
> > at
> > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> > va:167)
> > ... 7 more
> >
> > Best regards,
> > Henry
> >
> > ________________________________
> > The privileged confidential information contained in this email is
> > intended for use only by the addressees as indicated by the original
> > sender of this email. If you are not the addressee indicated in this
> > email or are not responsible for delivery of the email to such a
> > person, please kindly reply to the sender indicating this fact and
> > delete all copies of it from your computer and network server
> > immediately. Your cooperation is highly appreciated. It is advised
> > that any unauthorized use of confidential information of Winbond is
> > strictly prohibited; and any information in this email irrelevant to
> > the official business of Winbond shall be deemed as neither given nor
> endorsed by Winbond.
> >
>
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original sender
> of this email. If you are not the addressee indicated in this email or are
> not responsible for delivery of the email to such a person, please kindly
> reply to the sender indicating this fact and delete all copies of it from
> your computer and network server immediately. Your cooperation is highly
> appreciated. It is advised that any unauthorized use of confidential
> information of Winbond is strictly prohibited; and any information in this
> email irrelevant to the official business of Winbond shall be deemed as
> neither given nor endorsed by Winbond.
>

RE: scan + filter failed with OutOfOrderScannerNextException

[Henry Hung <YT...@winbond.com>]

Hi Anoop,

I receive this error in client side, and pretty sure the scan failed.
I'm using default caching, so it should be 100, right?
About scan time out period, I will try to set it higher, probably 1 hour.

BTW, I'm using hbase 0.96.0.

Best regards,
Henry

-----Original Message-----
From: Anoop John [mailto:anoop.hbase@gmail.com]
Sent: Monday, September 29, 2014 3:13 PM
To: user@hbase.apache.org
Subject: Re: scan + filter failed with OutOfOrderScannerNextException

Hi
   Even when the RS throws this Exception, the client side will start a new Scanner and retry. U just see this in log or the scan is failing altogether?
What is the caching you use on Scan?  When most of the rows are filtered out at server side, it takes more time to fetch and return the 'caching'
number of rows back to client. By this time the client would have already timed out and sent a retry. We allow such a retry will cause not getting back all the results. This is a serious problem and this Exception way
helps there.   You can try increasing the scan time out period at client
side. This should help.

-Anoop-

On Mon, Sep 29, 2014 at 12:35 PM, Henry Hung <YT...@winbond.com> wrote:

> Hi All,
>
> Is there a way to let scanner finish scanning all regions without
> throwing this kind of error? I'm using scan with filter MUST_PASS_ALL,
> and I observe that whenever the result data is smaller (let's say 10%)
> compare to another filter with larger result (let's say 80%), it always failed.
>
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> OutOfOrderScannerNextException: was there a rpc timeout?
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> 91)
> at
> com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> esPerfLogTimeResultSet.java:58)
> at
> com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M
> esPerfLogTimeResultSet.java:1) at
> com.winbond.hbase.executor.Program.main(Program.java:106)
> Caused by:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected
> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
> 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq:
> 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> java:3007)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> 2.callBlockingMethod(ClientProtos.java:26929)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> rAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> nstructorAccessorImpl.java:27) at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> tion.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> ption.java:95)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(Proto
> bufUtil.java:235)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> va:197)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> va:57)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> tryingCaller.java:116)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe
> tryingCaller.java:94)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3
> 43)
> ... 3 more
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected
> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
> 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq:
> 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.
> java:3007)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> 2.callBlockingMethod(ClientProtos.java:26929)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
>
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.jav
> a:1650)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation
> .callBlockingMethod(RpcClient.java:1708)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> BlockingStub.scan(ClientProtos.java:27332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja
> va:167)
> ... 7 more
>
> Best regards,
> Henry
>
> ________________________________
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original
> sender of this email. If you are not the addressee indicated in this
> email or are not responsible for delivery of the email to such a
> person, please kindly reply to the sender indicating this fact and
> delete all copies of it from your computer and network server
> immediately. Your cooperation is highly appreciated. It is advised
> that any unauthorized use of confidential information of Winbond is
> strictly prohibited; and any information in this email irrelevant to
> the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.
>

The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.

Re: scan + filter failed with OutOfOrderScannerNextException

[Anoop John <an...@gmail.com>]

Hi
   Even when the RS throws this Exception, the client side will start a new
Scanner and retry. U just see this in log or the scan is failing altogether?
What is the caching you use on Scan?  When most of the rows are filtered
out at server side, it takes more time to fetch and return the 'caching'
number of rows back to client. By this time the client would have already
timed out and sent a retry. We allow such a retry will cause not getting
back all the results. This is a serious problem and this Exception way
helps there.   You can try increasing the scan time out period at client
side. This should help.

-Anoop-

On Mon, Sep 29, 2014 at 12:35 PM, Henry Hung <YT...@winbond.com> wrote:

> Hi All,
>
> Is there a way to let scanner finish scanning all regions without throwing
> this kind of error? I'm using scan with filter MUST_PASS_ALL, and I observe
> that whenever the result data is smaller (let's say 10%) compare to another
> filter with larger result (let's say 80%), it always failed.
>
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> OutOfOrderScannerNextException: was there a rpc timeout?
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:391)
> at
> com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(MesPerfLogTimeResultSet.java:58)
> at
> com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(MesPerfLogTimeResultSet.java:1)
> at com.winbond.hbase.executor.Program.main(Program.java:106)
> Caused by:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
> 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq:
> 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3007)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:235)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:343)
> ... 3 more
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
> 1594996183743944787 number_of_rows: 100 close_scanner: false next_call_seq:
> 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3007)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
>
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:27332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:167)
> ... 7 more
>
> Best regards,
> Henry
>
> ________________________________
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original sender
> of this email. If you are not the addressee indicated in this email or are
> not responsible for delivery of the email to such a person, please kindly
> reply to the sender indicating this fact and delete all copies of it from
> your computer and network server immediately. Your cooperation is highly
> appreciated. It is advised that any unauthorized use of confidential
> information of Winbond is strictly prohibited; and any information in this
> email irrelevant to the official business of Winbond shall be deemed as
> neither given nor endorsed by Winbond.
>