You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vinay Gupta <vi...@yahoo.com.INVALID> on 2014/09/17 08:24:52 UTC
HBase 0.98.1 batch Increment throws OperationConflictException
>
> Hi,
> We are using Hbase batch API and with 0.98.1 we get the following exception on using batch() with Increment
> ————————————
> org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {5266048044724982303, 5395957753774586342} on row [rowkey13-20140331] may have already completed
> at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> at java.lang.Thread.run(Thread.java:745)
> ————————————
> Eventually the job fails with
> "Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
>
> The same job works in Hbase 0.94 installation. Any tips on which config settings to play with to resolve this?
> Is the application supposed to handle these exceptions? (something new in Hbase 0.98 or 0.96 ??)
>
> Thanks,
> -Vinay
>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Anoop John <an...@gmail.com>.
Yes that is also possible.. So in such a case this new behavior telling
the issue clearly. In the past the retry op would have silently succeeded
giving a wrong result overall!!!
-Anoop-
On Wed, Sep 17, 2014 at 7:14 PM, Vin Gup <vi...@yahoo.com.invalid>
wrote:
> Ok. I will try with your suggestions but I see this error even with
> batches with no row key duplicates. I still suspect that client is timing
> out and retrying too often and needs to back off as the region server is
> heavily loaded.
>
> -Vinay
>
> > On Sep 17, 2014, at 3:14 AM, Anoop John <an...@gmail.com> wrote:
> >
> > This is an improvement (rather an issue fix) done from 0.98+ versions.
> > This is for non-idempotent operations (like increment) which HBase
> clients
> > might retry on failure. Such retry can give wrong results (possibly
> > incrementing 2 times for one increment op)
> >
> > Can you change your application side code so as to avoid possible
> multiple
> > increment for same keys in one batch. Those can be combined to one
> > increment.
> >
> > You can turn off this improvement using config
> "hbase.client.nonces.enabled"
> > (Configure at client side xml file). But this is not a recommended way.
> U
> > can check with this.
> >
> > -Anoop-
> >
> >
> > On Wed, Sep 17, 2014 at 1:04 PM, Vin Gup <vi...@yahoo.com.invalid>
> > wrote:
> >
> >> Yes possibly. Why would that be a problem?
> >> Earlier client (0.94) didn't complain about it.
> >>
> >> Thanks,
> >> -Vinay
> >>
> >>> On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com>
> wrote:
> >>>
> >>> You have more than one increment for the same key in one batch?
> >>>
> >>> On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta
> >> <vi...@yahoo.com.invalid>
> >>> wrote:
> >>>
> >>>> Also the regionserver keeps throwing exceptions like
> >>>>
> >>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >>>> regionserver.ServerNonceManager: Conflict detected by nonce:
> >>>> [43871278468062569
> >>>> 89:2793719453824938427], [state 0, hasWait false, activity
> 06:55:41.091]
> >>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >>>> regionserver.ServerNonceManager: Conflict detected by nonce:
> >>>> [43871278468062569
> >>>> 89:843474753753473839], [state 0, hasWait false, activity
> 06:55:41.094]
> >>>>
> >>>>
> >>>> Are we sending data too fast? Is there a client side setting or a
> server
> >>>> side setting we need to look at to alleviate this?
> >>>> Again this was never a problem with HBase 0.94 cluster.
> >>>>
> >>>> We are calling the batch API in a List<> of 1000 increment and we do
> >>>> approx 30000 Increments (30 batches) at a time.
> >>>>
> >>>>
> >>>> -Vinay
> >>>>
> >>>>
> >>>> On Sep 16, 2014, at 11:24 PM, Vinay Gupta
> <vingup2005@yahoo.com.INVALID
> >>>
> >>>> wrote:
> >>>>
> >>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>> We are using Hbase batch API and with 0.98.1 we get the following
> >>>> exception on using batch() with Increment
> >>>>>> ————————————
> >>>>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
> >>>> operation with nonce {5266048044724982303, 5395957753774586342} on row
> >>>> [rowkey13-20140331] may have already completed
> >>>>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
> >>>>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> >>>>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> >>>>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> >>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>> ————————————
> >>>>>> Eventually the job fails with
> >>>>>> "Error:
> >>>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
> >>>>>>
> >>>>>> The same job works in Hbase 0.94 installation. Any tips on which
> >> config
> >>>> settings to play with to resolve this?
> >>>>>> Is the application supposed to handle these exceptions? (something
> new
> >>>> in Hbase 0.98 or 0.96 ??)
> >>>>>>
> >>>>>> Thanks,
> >>>>>> -Vinay
> >>
>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Vin Gup <vi...@yahoo.com.INVALID>.
Ok. I will try with your suggestions but I see this error even with batches with no row key duplicates. I still suspect that client is timing out and retrying too often and needs to back off as the region server is heavily loaded.
-Vinay
> On Sep 17, 2014, at 3:14 AM, Anoop John <an...@gmail.com> wrote:
>
> This is an improvement (rather an issue fix) done from 0.98+ versions.
> This is for non-idempotent operations (like increment) which HBase clients
> might retry on failure. Such retry can give wrong results (possibly
> incrementing 2 times for one increment op)
>
> Can you change your application side code so as to avoid possible multiple
> increment for same keys in one batch. Those can be combined to one
> increment.
>
> You can turn off this improvement using config "hbase.client.nonces.enabled"
> (Configure at client side xml file). But this is not a recommended way. U
> can check with this.
>
> -Anoop-
>
>
> On Wed, Sep 17, 2014 at 1:04 PM, Vin Gup <vi...@yahoo.com.invalid>
> wrote:
>
>> Yes possibly. Why would that be a problem?
>> Earlier client (0.94) didn't complain about it.
>>
>> Thanks,
>> -Vinay
>>
>>> On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com> wrote:
>>>
>>> You have more than one increment for the same key in one batch?
>>>
>>> On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta
>> <vi...@yahoo.com.invalid>
>>> wrote:
>>>
>>>> Also the regionserver keeps throwing exceptions like
>>>>
>>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>>>> regionserver.ServerNonceManager: Conflict detected by nonce:
>>>> [43871278468062569
>>>> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
>>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>>>> regionserver.ServerNonceManager: Conflict detected by nonce:
>>>> [43871278468062569
>>>> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
>>>>
>>>>
>>>> Are we sending data too fast? Is there a client side setting or a server
>>>> side setting we need to look at to alleviate this?
>>>> Again this was never a problem with HBase 0.94 cluster.
>>>>
>>>> We are calling the batch API in a List<> of 1000 increment and we do
>>>> approx 30000 Increments (30 batches) at a time.
>>>>
>>>>
>>>> -Vinay
>>>>
>>>>
>>>> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vingup2005@yahoo.com.INVALID
>>>
>>>> wrote:
>>>>
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>> We are using Hbase batch API and with 0.98.1 we get the following
>>>> exception on using batch() with Increment
>>>>>> ————————————
>>>>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
>>>> operation with nonce {5266048044724982303, 5395957753774586342} on row
>>>> [rowkey13-20140331] may have already completed
>>>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
>>>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
>>>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
>>>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
>>>>>> at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
>>>>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>>>>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>>>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>>>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>>>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> ————————————
>>>>>> Eventually the job fails with
>>>>>> "Error:
>>>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
>>>>>>
>>>>>> The same job works in Hbase 0.94 installation. Any tips on which
>> config
>>>> settings to play with to resolve this?
>>>>>> Is the application supposed to handle these exceptions? (something new
>>>> in Hbase 0.98 or 0.96 ??)
>>>>>>
>>>>>> Thanks,
>>>>>> -Vinay
>>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Anoop John <an...@gmail.com>.
This is an improvement (rather an issue fix) done from 0.98+ versions.
This is for non-idempotent operations (like increment) which HBase clients
might retry on failure. Such retry can give wrong results (possibly
incrementing 2 times for one increment op)
Can you change your application side code so as to avoid possible multiple
increment for same keys in one batch. Those can be combined to one
increment.
You can turn off this improvement using config "hbase.client.nonces.enabled"
(Configure at client side xml file). But this is not a recommended way. U
can check with this.
-Anoop-
On Wed, Sep 17, 2014 at 1:04 PM, Vin Gup <vi...@yahoo.com.invalid>
wrote:
> Yes possibly. Why would that be a problem?
> Earlier client (0.94) didn't complain about it.
>
> Thanks,
> -Vinay
>
> > On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com> wrote:
> >
> > You have more than one increment for the same key in one batch?
> >
> > On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta
> <vi...@yahoo.com.invalid>
> > wrote:
> >
> >> Also the regionserver keeps throwing exceptions like
> >>
> >> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >> regionserver.ServerNonceManager: Conflict detected by nonce:
> >> [43871278468062569
> >> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
> >> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >> regionserver.ServerNonceManager: Conflict detected by nonce:
> >> [43871278468062569
> >> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
> >>
> >>
> >> Are we sending data too fast? Is there a client side setting or a server
> >> side setting we need to look at to alleviate this?
> >> Again this was never a problem with HBase 0.94 cluster.
> >>
> >> We are calling the batch API in a List<> of 1000 increment and we do
> >> approx 30000 Increments (30 batches) at a time.
> >>
> >>
> >> -Vinay
> >>
> >>
> >> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vingup2005@yahoo.com.INVALID
> >
> >> wrote:
> >>
> >>>
> >>>>
> >>>> Hi,
> >>>> We are using Hbase batch API and with 0.98.1 we get the following
> >> exception on using batch() with Increment
> >>>> ————————————
> >>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
> >> operation with nonce {5266048044724982303, 5395957753774586342} on row
> >> [rowkey13-20140331] may have already completed
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
> >>>> at
> >>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
> >>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> >>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> ————————————
> >>>> Eventually the job fails with
> >>>> "Error:
> >> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
> >>>>
> >>>> The same job works in Hbase 0.94 installation. Any tips on which
> config
> >> settings to play with to resolve this?
> >>>> Is the application supposed to handle these exceptions? (something new
> >> in Hbase 0.98 or 0.96 ??)
> >>>>
> >>>> Thanks,
> >>>> -Vinay
> >>
> >>
>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Vinay Gupta <vi...@yahoo.com.INVALID>.
Actually we did not test data consistency issues in 0.94. So they might as well be there.
We don’t plan to turn the nonce feature off, btw.
Increasing hbase.rpc.timeout seems to solve this problem. My guess is client doesnt retry so often when we increase this value.
Another config which is supposed to directly control retry interval is hbase.client.pause. However we didn’t have much luck
with it.
-Vinay
On Sep 17, 2014, at 7:29 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. Earlier client (0.94) didn't complain about it.
>
> Did you observe any data loss (w.r.t. Increments) in 0.94 when the servers
> were loaded ?
>
> As Anoop said, it is not recommended to turn off this feature in 0.98
>
> On Wed, Sep 17, 2014 at 12:34 AM, Vin Gup <vi...@yahoo.com.invalid>
> wrote:
>
>> Yes possibly. Why would that be a problem?
>> Earlier client (0.94) didn't complain about it.
>>
>> Thanks,
>> -Vinay
>>
>>> On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com> wrote:
>>>
>>> You have more than one increment for the same key in one batch?
>>>
>>> On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta
>> <vi...@yahoo.com.invalid>
>>> wrote:
>>>
>>>> Also the regionserver keeps throwing exceptions like
>>>>
>>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>>>> regionserver.ServerNonceManager: Conflict detected by nonce:
>>>> [43871278468062569
>>>> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
>>>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>>>> regionserver.ServerNonceManager: Conflict detected by nonce:
>>>> [43871278468062569
>>>> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
>>>>
>>>>
>>>> Are we sending data too fast? Is there a client side setting or a server
>>>> side setting we need to look at to alleviate this?
>>>> Again this was never a problem with HBase 0.94 cluster.
>>>>
>>>> We are calling the batch API in a List<> of 1000 increment and we do
>>>> approx 30000 Increments (30 batches) at a time.
>>>>
>>>>
>>>> -Vinay
>>>>
>>>>
>>>> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vingup2005@yahoo.com.INVALID
>>>
>>>> wrote:
>>>>
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>> We are using Hbase batch API and with 0.98.1 we get the following
>>>> exception on using batch() with Increment
>>>>>> ————————————
>>>>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
>>>> operation with nonce {5266048044724982303, 5395957753774586342} on row
>>>> [rowkey13-20140331] may have already completed
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
>>>>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>>>>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>>>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> ————————————
>>>>>> Eventually the job fails with
>>>>>> "Error:
>>>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
>>>>>>
>>>>>> The same job works in Hbase 0.94 installation. Any tips on which
>> config
>>>> settings to play with to resolve this?
>>>>>> Is the application supposed to handle these exceptions? (something new
>>>> in Hbase 0.98 or 0.96 ??)
>>>>>>
>>>>>> Thanks,
>>>>>> -Vinay
>>>>
>>>>
>>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Ted Yu <yu...@gmail.com>.
bq. Earlier client (0.94) didn't complain about it.
Did you observe any data loss (w.r.t. Increments) in 0.94 when the servers
were loaded ?
As Anoop said, it is not recommended to turn off this feature in 0.98
On Wed, Sep 17, 2014 at 12:34 AM, Vin Gup <vi...@yahoo.com.invalid>
wrote:
> Yes possibly. Why would that be a problem?
> Earlier client (0.94) didn't complain about it.
>
> Thanks,
> -Vinay
>
> > On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com> wrote:
> >
> > You have more than one increment for the same key in one batch?
> >
> > On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta
> <vi...@yahoo.com.invalid>
> > wrote:
> >
> >> Also the regionserver keeps throwing exceptions like
> >>
> >> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >> regionserver.ServerNonceManager: Conflict detected by nonce:
> >> [43871278468062569
> >> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
> >> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> >> regionserver.ServerNonceManager: Conflict detected by nonce:
> >> [43871278468062569
> >> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
> >>
> >>
> >> Are we sending data too fast? Is there a client side setting or a server
> >> side setting we need to look at to alleviate this?
> >> Again this was never a problem with HBase 0.94 cluster.
> >>
> >> We are calling the batch API in a List<> of 1000 increment and we do
> >> approx 30000 Increments (30 batches) at a time.
> >>
> >>
> >> -Vinay
> >>
> >>
> >> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vingup2005@yahoo.com.INVALID
> >
> >> wrote:
> >>
> >>>
> >>>>
> >>>> Hi,
> >>>> We are using Hbase batch API and with 0.98.1 we get the following
> >> exception on using batch() with Increment
> >>>> ————————————
> >>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
> >> operation with nonce {5266048044724982303, 5395957753774586342} on row
> >> [rowkey13-20140331] may have already completed
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
> >>>> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
> >>>> at
> >>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
> >>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> >>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> >>>> at
> >>
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> ————————————
> >>>> Eventually the job fails with
> >>>> "Error:
> >> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
> >>>>
> >>>> The same job works in Hbase 0.94 installation. Any tips on which
> config
> >> settings to play with to resolve this?
> >>>> Is the application supposed to handle these exceptions? (something new
> >> in Hbase 0.98 or 0.96 ??)
> >>>>
> >>>> Thanks,
> >>>> -Vinay
> >>
> >>
>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Vin Gup <vi...@yahoo.com.INVALID>.
Yes possibly. Why would that be a problem?
Earlier client (0.94) didn't complain about it.
Thanks,
-Vinay
> On Sep 17, 2014, at 12:16 AM, Anoop John <an...@gmail.com> wrote:
>
> You have more than one increment for the same key in one batch?
>
> On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta <vi...@yahoo.com.invalid>
> wrote:
>
>> Also the regionserver keeps throwing exceptions like
>>
>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>> regionserver.ServerNonceManager: Conflict detected by nonce:
>> [43871278468062569
>> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
>> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
>> regionserver.ServerNonceManager: Conflict detected by nonce:
>> [43871278468062569
>> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
>>
>>
>> Are we sending data too fast? Is there a client side setting or a server
>> side setting we need to look at to alleviate this?
>> Again this was never a problem with HBase 0.94 cluster.
>>
>> We are calling the batch API in a List<> of 1000 increment and we do
>> approx 30000 Increments (30 batches) at a time.
>>
>>
>> -Vinay
>>
>>
>> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vi...@yahoo.com.INVALID>
>> wrote:
>>
>>>
>>>>
>>>> Hi,
>>>> We are using Hbase batch API and with 0.98.1 we get the following
>> exception on using batch() with Increment
>>>> ————————————
>>>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
>> operation with nonce {5266048044724982303, 5395957753774586342} on row
>> [rowkey13-20140331] may have already completed
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
>>>> at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
>>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>>>> at
>> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> ————————————
>>>> Eventually the job fails with
>>>> "Error:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
>>>>
>>>> The same job works in Hbase 0.94 installation. Any tips on which config
>> settings to play with to resolve this?
>>>> Is the application supposed to handle these exceptions? (something new
>> in Hbase 0.98 or 0.96 ??)
>>>>
>>>> Thanks,
>>>> -Vinay
>>
>>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Anoop John <an...@gmail.com>.
You have more than one increment for the same key in one batch?
On Wed, Sep 17, 2014 at 12:33 PM, Vinay Gupta <vi...@yahoo.com.invalid>
wrote:
> Also the regionserver keeps throwing exceptions like
>
> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> regionserver.ServerNonceManager: Conflict detected by nonce:
> [43871278468062569
> 89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
> 2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020]
> regionserver.ServerNonceManager: Conflict detected by nonce:
> [43871278468062569
> 89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
>
>
> Are we sending data too fast? Is there a client side setting or a server
> side setting we need to look at to alleviate this?
> Again this was never a problem with HBase 0.94 cluster.
>
> We are calling the batch API in a List<> of 1000 increment and we do
> approx 30000 Increments (30 batches) at a time.
>
>
> -Vinay
>
>
> On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vi...@yahoo.com.INVALID>
> wrote:
>
> >
> >>
> >> Hi,
> >> We are using Hbase batch API and with 0.98.1 we get the following
> exception on using batch() with Increment
> >> ————————————
> >> org.apache.hadoop.hbase.exceptions.OperationConflictException: The
> operation with nonce {5266048044724982303, 5395957753774586342} on row
> [rowkey13-20140331] may have already completed
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
> >> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> >> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> >> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> >> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> >> at java.lang.Thread.run(Thread.java:745)
> >> ————————————
> >> Eventually the job fails with
> >> "Error:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
> >>
> >> The same job works in Hbase 0.94 installation. Any tips on which config
> settings to play with to resolve this?
> >> Is the application supposed to handle these exceptions? (something new
> in Hbase 0.98 or 0.96 ??)
> >>
> >> Thanks,
> >> -Vinay
> >>
> >
>
>
Re: HBase 0.98.1 batch Increment throws OperationConflictException
Posted by Vinay Gupta <vi...@yahoo.com.INVALID>.
Also the regionserver keeps throwing exceptions like
2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020] regionserver.ServerNonceManager: Conflict detected by nonce: [43871278468062569
89:2793719453824938427], [state 0, hasWait false, activity 06:55:41.091]
2014-09-17 06:56:07,151 DEBUG [RpcServer.handler=10,port=60020] regionserver.ServerNonceManager: Conflict detected by nonce: [43871278468062569
89:843474753753473839], [state 0, hasWait false, activity 06:55:41.094]
Are we sending data too fast? Is there a client side setting or a server side setting we need to look at to alleviate this?
Again this was never a problem with HBase 0.94 cluster.
We are calling the batch API in a List<> of 1000 increment and we do approx 30000 Increments (30 batches) at a time.
-Vinay
On Sep 16, 2014, at 11:24 PM, Vinay Gupta <vi...@yahoo.com.INVALID> wrote:
>
>>
>> Hi,
>> We are using Hbase batch API and with 0.98.1 we get the following exception on using batch() with Increment
>> ————————————
>> org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {5266048044724982303, 5395957753774586342} on row [rowkey13-20140331] may have already completed
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4199)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4163)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3424)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>> at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>> at java.lang.Thread.run(Thread.java:745)
>> ————————————
>> Eventually the job fails with
>> "Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException"
>>
>> The same job works in Hbase 0.94 installation. Any tips on which config settings to play with to resolve this?
>> Is the application supposed to handle these exceptions? (something new in Hbase 0.98 or 0.96 ??)
>>
>> Thanks,
>> -Vinay
>>
>