You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by anupama agarwal <an...@gmail.com> on 2016/06/15 06:19:45 UTC

Dropping of Index can still leave some non-replayed writes Phoenix-2915

Hi All,

I have hit this error in phoenix, Phoenix-2915. It could be possible , that
there are some index writes in WAL which are not replayed and the index is
dropped.

And, now the table is not there, these writes cannot be replayed which
result in data table regions also to not come up. My data region is in
FAILED_TO_OPEN state. I have tried recreating the index, and still region
is not able to come up. I realise that this has been fixed in new version
of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
suggest a solution?

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by anupama agarwal <an...@gmail.com>.
This didn't work Ankit. Please find below detailed logs on region server.
Please let me know what else I can try. When I try to run hbck command,
this is the error I get:

ERROR: There is a hole in the region chain between \x06 and \x07.  You need
to create a new .regioninfo and region dir in hdfs to plug the hole.


2016-06-15 21:53:41,132 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"totalColumns":25,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"_0","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_GROUP","vlen":8},{"timestamp":1465911182647,"tag":[],"qualifier":"0:EXTERNAL_REF_ID","vlen":10},{"timestamp":1465911182647,"tag":[],"qualifier":"0:PARTY_ID_FROM","vlen":4}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,140 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"ts":1465911182647,"totalColumns":12,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_6","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_10","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_9","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_8","vlen":0}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,216 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of
region=APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.,
starting to roll back the global memstore size.

org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
Failed to write to multiple index tables

at
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:220)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:169)

at org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:545)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1422)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1663)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1738)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1695)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1413)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3940)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3797)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:969)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:841)

at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:814)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5828)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5794)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)

at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2016-06-15 21:53:41,220 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
region {ENCODED => eaee593f66a1ef282a2b5ae352624982, NAME =>
'APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.',
STARTKEY => '\x06', ENDKEY => '\x07'} failed, transitioning from OPENING to
FAILED_OPEN in ZK, expecting version 22

2016-06-15 21:57:50,092 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=8.37 MB,
freeSize=7.97 GB, max=7.98 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0



On Wed, Jun 15, 2016 at 8:48 PM, Ankit Singhal <an...@gmail.com>
wrote:

> Yes, restart your cluster
>
> On Wed, Jun 15, 2016 at 8:17 AM, anupama agarwal <an...@gmail.com>
> wrote:
>
>> I have created async index with same name. But I am still getting the
>> same error. Should I restart my cluster for changes to reflect?
>> On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com>
>> wrote:
>>
>>> Hi Anupama,
>>>
>>> Option 1:-
>>> You can create a ASYNC index so that WAL can be replayed. And once your
>>> regions are up , remember to do the flush of data table before dropping the
>>> index.
>>>
>>> Option 2:-
>>> Create a table in hbase with the same name as index table name by using
>>> hbase shell.
>>>
>>> Regards,
>>> Ankit Singhal
>>>
>>>
>>> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>>>> that there are some index writes in WAL which are not replayed and the
>>>> index is dropped.
>>>>
>>>> And, now the table is not there, these writes cannot be replayed which
>>>> result in data table regions also to not come up. My data region is in
>>>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>>>> is not able to come up. I realise that this has been fixed in new version
>>>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>>>> suggest a solution?
>>>>
>>>
>>>
>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by anupama agarwal <an...@gmail.com>.
This didn't work Ankit. Please find below detailed logs on region server.
Please let me know what else I can try. When I try to run hbck command,
this is the error I get:

ERROR: There is a hole in the region chain between \x06 and \x07.  You need
to create a new .regioninfo and region dir in hdfs to plug the hole.


2016-06-15 21:53:41,132 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"totalColumns":25,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"_0","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_GROUP","vlen":8},{"timestamp":1465911182647,"tag":[],"qualifier":"0:EXTERNAL_REF_ID","vlen":10},{"timestamp":1465911182647,"tag":[],"qualifier":"0:PARTY_ID_FROM","vlen":4}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,140 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"ts":1465911182647,"totalColumns":12,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_6","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_10","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_9","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_8","vlen":0}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,216 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of
region=APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.,
starting to roll back the global memstore size.

org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
Failed to write to multiple index tables

at
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:220)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:169)

at org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:545)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1422)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1663)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1738)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1695)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1413)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3940)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3797)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:969)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:841)

at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:814)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5828)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5794)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)

at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2016-06-15 21:53:41,220 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
region {ENCODED => eaee593f66a1ef282a2b5ae352624982, NAME =>
'APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.',
STARTKEY => '\x06', ENDKEY => '\x07'} failed, transitioning from OPENING to
FAILED_OPEN in ZK, expecting version 22

2016-06-15 21:57:50,092 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=8.37 MB,
freeSize=7.97 GB, max=7.98 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0



On Wed, Jun 15, 2016 at 8:48 PM, Ankit Singhal <an...@gmail.com>
wrote:

> Yes, restart your cluster
>
> On Wed, Jun 15, 2016 at 8:17 AM, anupama agarwal <an...@gmail.com>
> wrote:
>
>> I have created async index with same name. But I am still getting the
>> same error. Should I restart my cluster for changes to reflect?
>> On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com>
>> wrote:
>>
>>> Hi Anupama,
>>>
>>> Option 1:-
>>> You can create a ASYNC index so that WAL can be replayed. And once your
>>> regions are up , remember to do the flush of data table before dropping the
>>> index.
>>>
>>> Option 2:-
>>> Create a table in hbase with the same name as index table name by using
>>> hbase shell.
>>>
>>> Regards,
>>> Ankit Singhal
>>>
>>>
>>> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>>>> that there are some index writes in WAL which are not replayed and the
>>>> index is dropped.
>>>>
>>>> And, now the table is not there, these writes cannot be replayed which
>>>> result in data table regions also to not come up. My data region is in
>>>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>>>> is not able to come up. I realise that this has been fixed in new version
>>>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>>>> suggest a solution?
>>>>
>>>
>>>
>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by Ankit Singhal <an...@gmail.com>.
Yes, restart your cluster

On Wed, Jun 15, 2016 at 8:17 AM, anupama agarwal <an...@gmail.com> wrote:

> I have created async index with same name. But I am still getting the same
> error. Should I restart my cluster for changes to reflect?
> On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com> wrote:
>
>> Hi Anupama,
>>
>> Option 1:-
>> You can create a ASYNC index so that WAL can be replayed. And once your
>> regions are up , remember to do the flush of data table before dropping the
>> index.
>>
>> Option 2:-
>> Create a table in hbase with the same name as index table name by using
>> hbase shell.
>>
>> Regards,
>> Ankit Singhal
>>
>>
>> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>>> that there are some index writes in WAL which are not replayed and the
>>> index is dropped.
>>>
>>> And, now the table is not there, these writes cannot be replayed which
>>> result in data table regions also to not come up. My data region is in
>>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>>> is not able to come up. I realise that this has been fixed in new version
>>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>>> suggest a solution?
>>>
>>
>>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by Ankit Singhal <an...@gmail.com>.
Yes, restart your cluster

On Wed, Jun 15, 2016 at 8:17 AM, anupama agarwal <an...@gmail.com> wrote:

> I have created async index with same name. But I am still getting the same
> error. Should I restart my cluster for changes to reflect?
> On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com> wrote:
>
>> Hi Anupama,
>>
>> Option 1:-
>> You can create a ASYNC index so that WAL can be replayed. And once your
>> regions are up , remember to do the flush of data table before dropping the
>> index.
>>
>> Option 2:-
>> Create a table in hbase with the same name as index table name by using
>> hbase shell.
>>
>> Regards,
>> Ankit Singhal
>>
>>
>> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>>> that there are some index writes in WAL which are not replayed and the
>>> index is dropped.
>>>
>>> And, now the table is not there, these writes cannot be replayed which
>>> result in data table regions also to not come up. My data region is in
>>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>>> is not able to come up. I realise that this has been fixed in new version
>>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>>> suggest a solution?
>>>
>>
>>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by anupama agarwal <an...@gmail.com>.
I have created async index with same name. But I am still getting the same
error. Should I restart my cluster for changes to reflect?
On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com> wrote:

> Hi Anupama,
>
> Option 1:-
> You can create a ASYNC index so that WAL can be replayed. And once your
> regions are up , remember to do the flush of data table before dropping the
> index.
>
> Option 2:-
> Create a table in hbase with the same name as index table name by using
> hbase shell.
>
> Regards,
> Ankit Singhal
>
>
> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>> that there are some index writes in WAL which are not replayed and the
>> index is dropped.
>>
>> And, now the table is not there, these writes cannot be replayed which
>> result in data table regions also to not come up. My data region is in
>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>> is not able to come up. I realise that this has been fixed in new version
>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>> suggest a solution?
>>
>
>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by anupama agarwal <an...@gmail.com>.
I have created async index with same name. But I am still getting the same
error. Should I restart my cluster for changes to reflect?
On Jun 15, 2016 8:38 PM, "Ankit Singhal" <an...@gmail.com> wrote:

> Hi Anupama,
>
> Option 1:-
> You can create a ASYNC index so that WAL can be replayed. And once your
> regions are up , remember to do the flush of data table before dropping the
> index.
>
> Option 2:-
> Create a table in hbase with the same name as index table name by using
> hbase shell.
>
> Regards,
> Ankit Singhal
>
>
> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>> that there are some index writes in WAL which are not replayed and the
>> index is dropped.
>>
>> And, now the table is not there, these writes cannot be replayed which
>> result in data table regions also to not come up. My data region is in
>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>> is not able to come up. I realise that this has been fixed in new version
>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>> suggest a solution?
>>
>
>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by Ankit Singhal <an...@gmail.com>.
Hi Anupama,

Option 1:-
You can create a ASYNC index so that WAL can be replayed. And once your
regions are up , remember to do the flush of data table before dropping the
index.

Option 2:-
Create a table in hbase with the same name as index table name by using
hbase shell.

Regards,
Ankit Singhal


On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com> wrote:

> Hi All,
>
> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
> that there are some index writes in WAL which are not replayed and the
> index is dropped.
>
> And, now the table is not there, these writes cannot be replayed which
> result in data table regions also to not come up. My data region is in
> FAILED_TO_OPEN state. I have tried recreating the index, and still region
> is not able to come up. I realise that this has been fixed in new version
> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
> suggest a solution?
>

Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915

Posted by Ankit Singhal <an...@gmail.com>.
Hi Anupama,

Option 1:-
You can create a ASYNC index so that WAL can be replayed. And once your
regions are up , remember to do the flush of data table before dropping the
index.

Option 2:-
Create a table in hbase with the same name as index table name by using
hbase shell.

Regards,
Ankit Singhal


On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <an...@gmail.com> wrote:

> Hi All,
>
> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
> that there are some index writes in WAL which are not replayed and the
> index is dropped.
>
> And, now the table is not there, these writes cannot be replayed which
> result in data table regions also to not come up. My data region is in
> FAILED_TO_OPEN state. I have tried recreating the index, and still region
> is not able to come up. I realise that this has been fixed in new version
> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
> suggest a solution?
>