Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Francis Chuang <fr...@apache.org>]

Hi All,

I recently updated one of my Go apps to use Phoenix 5.0 with HBase 
2.0.2. I am using my Phoenix + HBase all in one docker image available 
here: https://github.com/Boostport/hbase-phoenix-all-in-one

This is the log/output from the exception:

RuntimeException: org.apache.phoenix.execute.CommitException: 
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 1 action: 
org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: 
Failed to build index for unexpected reason!
             at 
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
             at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
             at 
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
             at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
         Caused by: java.lang.VerifyError: class 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides 
final method 
compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
             at java.lang.ClassLoader.defineClass1(Native Method)
             at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
             at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
             at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
             at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
             at java.security.AccessController.doPrivileged(Native Method)
             at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
             at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
             at 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
             at 
org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
             at 
org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
             at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
             ... 18 more
         : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341 
-> CommitException: 
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 1 action: 
org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: 
Failed to build index for unexpected reason!
             at 
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
             at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
             at 
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
             at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
         Caused by: java.lang.VerifyError: class 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides 
final method 
compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
             at java.lang.ClassLoader.defineClass1(Native Method)
             at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
             at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
             at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
             at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
             at java.security.AccessController.doPrivileged(Native Method)
             at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
             at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
             at 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
             at 
org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
             at 
org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
             at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
             ... 18 more
         : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341 
-> RetriesExhaustedWithDetailsException: Failed 1 action: 
org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: 
Failed to build index for unexpected reason!
             at 
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
             at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
             at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
             at 
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
             at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
             at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
             at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
             at 
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
             at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
             at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
         Caused by: java.lang.VerifyError: class 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides 
final method 
compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
             at java.lang.ClassLoader.defineClass1(Native Method)
             at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
             at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
             at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
             at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
             at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
             at java.security.AccessController.doPrivileged(Native Method)
             at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
             at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
             at 
org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
             at 
org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
             at 
org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
             at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
             at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
             ... 18 more
         : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341

I am not entirely sure if this is a bug, so have not opened a JIRA yet. 
Due to Guava issues with Tephra (see TEPHRA-181), I replaced HBase's 
Guava 11.0.2 with Guava 13.0.1. See 
https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43

Is Guava 13.0.1 compatible with this release (signaling that this might 
be cause by incompatible versions of Guava) or should I be investigating 
something else?

Francis

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Josh Elser <el...@apache.org>]

Your assumptions are not unreasonable :) Phoenix 5.0.x should certainly 
work with HBase 2.0.x. Glad to see that it's been corrected already 
(embarassing that I don't even remember reviewing this).

Let me start a thread on dev@phoenix about a 5.0.1 or a 5.1.0. We need 
to have a Phoenix 5.x that works with all of HBase 2.0.x (and hopefully 
2.1.x too).


On 9/25/18 9:25 PM, Francis Chuang wrote:
> After some investigation, I found that Phoenix 5.0.0 is only compatible 
> with HBase 2.0.0.
> 
> In 2.0.1 and onward, compare(final Cell a, final Cell b) in 
> CellComparatorImpl was changed to final: 
> https://github.com/apache/hbase/blame/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java#L67
> 
> This change affected HBase 2.0.1 and 2.0.2.
> 
> As Phoenix 5.0.0 relies on this behavior: 
> https://github.com/apache/phoenix/blob/8a819c6c3b4befce190c6ac759f744df511de61d/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/covered/data/IndexMemStore.java#L84
> 
> Fortunately, this is fixed in Phoenix master: 
> https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/covered/data/IndexMemStore.java#L82
> 
> The issue should be resolved in the next release of Phoenix.
> 
> The problem is that I wrongly assumed HBase's version numbers to be 
> following semver and that a patch release would not introduce breaking 
> changes.
> 
> On 26/09/2018 1:04 AM, Jaanai Zhang wrote:
>>
>>
>>     Is my method of installing HBase and Phoenix correct?
>>
>> Did you check versions of HBase that exists in your classpath?
>>
>>     Is this a compatibility issue with Guava?
>>
>> It isn't an exception which incompatible with Guava
>>
>> ----------------------------------------
>>    Jaanai Zhang
>>    Best regards!
>>
>>
>>
>> Francis Chuang <francischuang@apache.org 
>> <ma...@apache.org>> 于2018年9月25日周二 下午8:25写道：
>>
>>     Thanks for taking a look, Jaanai!
>>
>>     Is my method of installing HBase and Phoenix correct? See
>>     https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L12
>>
>>     Is this a compatibility issue with Guava?
>>
>>     Francis
>>
>>     On 25/09/2018 10:21 PM, Jaanai Zhang wrote:
>>>
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>>         overrides
>>>         final method
>>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>>                      at
>>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>                      at
>>>
>>>     It looks like that HBase's Jars are incompatible.
>>>
>>>     ----------------------------------------
>>>        Jaanai Zhang
>>>        Best regards!
>>>
>>>
>>>
>>>     Francis Chuang <francischuang@apache.org
>>>     <ma...@apache.org>> 于2018年9月25日周二 下午8:06写道：
>>>
>>>         Hi All,
>>>
>>>         I recently updated one of my Go apps to use Phoenix 5.0 with
>>>         HBase
>>>         2.0.2. I am using my Phoenix + HBase all in one docker image
>>>         available
>>>         here: https://github.com/Boostport/hbase-phoenix-all-in-one
>>>
>>>         This is the log/output from the exception:
>>>
>>>         RuntimeException: org.apache.phoenix.execute.CommitException:
>>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>>
>>>         Failed 1 action:
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>>
>>>         Failed to build index for unexpected reason!
>>>                      at
>>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>>                      at
>>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>>                  Caused by: java.lang.VerifyError: class
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>>         overrides
>>>         final method
>>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>>                      at
>>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>                      at
>>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>                      at
>>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>                      at
>>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>                      at
>>>         java.security.AccessController.doPrivileged(Native Method)
>>>                      at
>>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>                      at
>>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>>                      at
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>>                      ... 18 more
>>>                  : 1 time, servers with issues:
>>>         9ac923bd5c9f,16020,1537875547341
>>>         -> CommitException:
>>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>>
>>>         Failed 1 action:
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>>
>>>         Failed to build index for unexpected reason!
>>>                      at
>>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>>                      at
>>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>>                  Caused by: java.lang.VerifyError: class
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>>         overrides
>>>         final method
>>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>>                      at
>>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>                      at
>>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>                      at
>>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>                      at
>>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>                      at
>>>         java.security.AccessController.doPrivileged(Native Method)
>>>                      at
>>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>                      at
>>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>>                      at
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>>                      ... 18 more
>>>                  : 1 time, servers with issues:
>>>         9ac923bd5c9f,16020,1537875547341
>>>         -> RetriesExhaustedWithDetailsException: Failed 1 action:
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>>
>>>         Failed to build index for unexpected reason!
>>>                      at
>>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>>                      at
>>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>>                      at
>>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>>                      at
>>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>>                      at
>>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>>                  Caused by: java.lang.VerifyError: class
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>>         overrides
>>>         final method
>>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>>                      at
>>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>                      at
>>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>                      at
>>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>                      at
>>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>                      at
>>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>                      at
>>>         java.security.AccessController.doPrivileged(Native Method)
>>>                      at
>>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>                      at
>>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>>                      at
>>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>>                      at
>>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>>                      at
>>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>>                      at
>>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>>                      ... 18 more
>>>                  : 1 time, servers with issues:
>>>         9ac923bd5c9f,16020,1537875547341
>>>
>>>         I am not entirely sure if this is a bug, so have not opened a
>>>         JIRA yet.
>>>         Due to Guava issues with Tephra (see TEPHRA-181), I replaced
>>>         HBase's
>>>         Guava 11.0.2 with Guava 13.0.1. See
>>>         https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>>>
>>>         Is Guava 13.0.1 compatible with this release (signaling that
>>>         this might
>>>         be cause by incompatible versions of Guava) or should I be
>>>         investigating
>>>         something else?
>>>
>>>         Francis
>>>
>>
> 

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Francis Chuang <fr...@apache.org>]

After some investigation, I found that Phoenix 5.0.0 is only compatible 
with HBase 2.0.0.

In 2.0.1 and onward, compare(final Cell a, final Cell b) in 
CellComparatorImpl was changed to final: 
https://github.com/apache/hbase/blame/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java#L67

This change affected HBase 2.0.1 and 2.0.2.

As Phoenix 5.0.0 relies on this behavior: 
https://github.com/apache/phoenix/blob/8a819c6c3b4befce190c6ac759f744df511de61d/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/covered/data/IndexMemStore.java#L84

Fortunately, this is fixed in Phoenix master: 
https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/covered/data/IndexMemStore.java#L82

The issue should be resolved in the next release of Phoenix.

The problem is that I wrongly assumed HBase's version numbers to be 
following semver and that a patch release would not introduce breaking 
changes.

On 26/09/2018 1:04 AM, Jaanai Zhang wrote:
>
>
>     Is my method of installing HBase and Phoenix correct?
>
> Did you check versions of HBase that exists in your classpath?
>
>     Is this a compatibility issue with Guava?
>
> It isn't an exception which incompatible with Guava
>
> ----------------------------------------
>    Jaanai Zhang
>    Best regards!
>
>
>
> Francis Chuang <francischuang@apache.org 
> <ma...@apache.org>> 于2018年9月25日周二 下午8:25写道：
>
>     Thanks for taking a look, Jaanai!
>
>     Is my method of installing HBase and Phoenix correct? See
>     https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L12
>
>     Is this a compatibility issue with Guava?
>
>     Francis
>
>     On 25/09/2018 10:21 PM, Jaanai Zhang wrote:
>>
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>
>>     It looks like that HBase's Jars are incompatible.
>>
>>     ----------------------------------------
>>        Jaanai Zhang
>>        Best regards!
>>
>>
>>
>>     Francis Chuang <francischuang@apache.org
>>     <ma...@apache.org>> 于2018年9月25日周二 下午8:06写道：
>>
>>         Hi All,
>>
>>         I recently updated one of my Go apps to use Phoenix 5.0 with
>>         HBase
>>         2.0.2. I am using my Phoenix + HBase all in one docker image
>>         available
>>         here: https://github.com/Boostport/hbase-phoenix-all-in-one
>>
>>         This is the log/output from the exception:
>>
>>         RuntimeException: org.apache.phoenix.execute.CommitException:
>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>
>>         Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>         -> CommitException:
>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>
>>         Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>         -> RetriesExhaustedWithDetailsException: Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>
>>         I am not entirely sure if this is a bug, so have not opened a
>>         JIRA yet.
>>         Due to Guava issues with Tephra (see TEPHRA-181), I replaced
>>         HBase's
>>         Guava 11.0.2 with Guava 13.0.1. See
>>         https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>>
>>         Is Guava 13.0.1 compatible with this release (signaling that
>>         this might
>>         be cause by incompatible versions of Guava) or should I be
>>         investigating
>>         something else?
>>
>>         Francis
>>
>

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Francis Chuang <fr...@apache.org>]

Thank you for your help!

I am not very familiar with Java, but here's what I did:
- "echo $CLASSPATH" did not return anything.

I then tried the following:
- export HBASE_CP=/opt/hbase/lib
- export HBASE_CONF_DIR=/opt/hbase/conf
- export HBASE_HOME=/opt/hbase
- /opt/hbase/bin/tephra classpath

The classpath returned is:

/opt/hbase/bin/../lib/*:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-client/target/*:/opt/hbase/conf:/usr/lib/jvm/java-1.8-openjdk/jre/lib/tools.jar:/opt/hbase:/opt/hbase/lib/aopalliance-1.0.jar:/opt/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/lib/api-util-1.0.0-M20.jar:/opt/hbase/lib/asm-3.1.jar:/opt/hbase/lib/audience-annotations-0.5.0.jar:/opt/hbase/lib/avro-1.7.7.jar:/opt/hbase/lib/commons-beanutils-core-1.8.0.jar:/opt/hbase/lib/commons-cli-1.2.jar:/opt/hbase/lib/commons-codec-1.10.jar:/opt/hbase/lib/commons-collections-3.2.2.jar:/opt/hbase/lib/commons-compress-1.4.1.jar:/opt/hbase/lib/commons-configuration-1.6.jar:/opt/hbase/lib/commons-crypto-1.0.0.jar:/opt/hbase/lib/commons-daemon-1.0.13.jar:/opt/hbase/lib/commons-digester-1.8.jar:/opt/hbase/lib/commons-httpclient-3.1.jar:/opt/hbase/lib/commons-io-2.5.jar:/opt/hbase/lib/commons-lang-2.6.jar:/opt/hbase/lib/commons-lang3-3.6.jar:/opt/hbase/lib/commons-logging-1.2.jar:/opt/hbase/lib/commons-math3-3.6.1.jar:/opt/hbase/lib/commons-net-3.1.jar:/opt/hbase/lib/curator-client-4.0.0.jar:/opt/hbase/lib/curator-framework-4.0.0.jar:/opt/hbase/lib/curator-recipes-4.0.0.jar:/opt/hbase/lib/disruptor-3.3.6.jar:/opt/hbase/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/lib/gson-2.2.4.jar:/opt/hbase/lib/guice-3.0.jar:/opt/hbase/lib/guice-servlet-3.0.jar:/opt/hbase/lib/hadoop-annotations-2.7.7.jar:/opt/hbase/lib/hadoop-auth-2.7.7.jar:/opt/hbase/lib/hadoop-client-2.7.7.jar:/opt/hbase/lib/hadoop-common-2.7.7-tests.jar:/opt/hbase/lib/hadoop-common-2.7.7.jar:/opt/hbase/lib/hadoop-distcp-2.7.7.jar:/opt/hbase/lib/hadoop-hdfs-2.7.7-tests.jar:/opt/hbase/lib/hadoop-hdfs-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-app-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-common-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-core-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-hs-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-jobclient-2.7.7.jar:/opt/hbase/lib/hadoop-mapreduce-client-shuffle-2.7.7.jar:/opt/hbase/lib/hadoop-minicluster-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-api-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-client-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-common-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-server-applicationhistoryservice-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-server-common-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-server-nodemanager-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-server-resourcemanager-2.7.7.jar:/opt/hbase/lib/hadoop-yarn-server-tests-2.7.7-tests.jar:/opt/hbase/lib/hadoop-yarn-server-web-proxy-2.7.7.jar:/opt/hbase/lib/hamcrest-core-1.3.jar:/opt/hbase/lib/hbase-annotations-2.0.2-tests.jar:/opt/hbase/lib/hbase-annotations-2.0.2.jar:/opt/hbase/lib/hbase-client-2.0.2.jar:/opt/hbase/lib/hbase-common-2.0.2-tests.jar:/opt/hbase/lib/hbase-common-2.0.2.jar:/opt/hbase/lib/hbase-endpoint-2.0.2.jar:/opt/hbase/lib/hbase-examples-2.0.2.jar:/opt/hbase/lib/hbase-external-blockcache-2.0.2.jar:/opt/hbase/lib/hbase-hadoop-compat-2.0.2-tests.jar:/opt/hbase/lib/hbase-hadoop-compat-2.0.2.jar:/opt/hbase/lib/hbase-hadoop2-compat-2.0.2-tests.jar:/opt/hbase/lib/hbase-hadoop2-compat-2.0.2.jar:/opt/hbase/lib/hbase-http-2.0.2.jar:/opt/hbase/lib/hbase-it-2.0.2-tests.jar:/opt/hbase/lib/hbase-it-2.0.2.jar:/opt/hbase/lib/hbase-mapreduce-2.0.2-tests.jar:/opt/hbase/lib/hbase-mapreduce-2.0.2.jar:/opt/hbase/lib/hbase-metrics-2.0.2.jar:/opt/hbase/lib/hbase-metrics-api-2.0.2.jar:/opt/hbase/lib/hbase-procedure-2.0.2.jar:/opt/hbase/lib/hbase-protocol-2.0.2.jar:/opt/hbase/lib/hbase-protocol-shaded-2.0.2.jar:/opt/hbase/lib/hbase-replication-2.0.2.jar:/opt/hbase/lib/hbase-resource-bundle-2.0.2.jar:/opt/hbase/lib/hbase-rest-2.0.2.jar:/opt/hbase/lib/hbase-rsgroup-2.0.2-tests.jar:/opt/hbase/lib/hbase-rsgroup-2.0.2.jar:/opt/hbase/lib/hbase-server-2.0.2-tests.jar:/opt/hbase/lib/hbase-server-2.0.2.jar:/opt/hbase/lib/hbase-shaded-miscellaneous-2.1.0.jar:/opt/hbase/lib/hbase-shaded-netty-2.1.0.jar:/opt/hbase/lib/hbase-shaded-protobuf-2.1.0.jar:/opt/hbase/lib/hbase-shell-2.0.2.jar:/opt/hbase/lib/hbase-testing-util-2.0.2.jar:/opt/hbase/lib/hbase-thrift-2.0.2.jar:/opt/hbase/lib/hbase-zookeeper-2.0.2-tests.jar:/opt/hbase/lib/hbase-zookeeper-2.0.2.jar:/opt/hbase/lib/hk2-api-2.5.0-b32.jar:/opt/hbase/lib/hk2-locator-2.5.0-b32.jar:/opt/hbase/lib/hk2-utils-2.5.0-b32.jar:/opt/hbase/lib/htrace-core-3.2.0-incubating.jar:/opt/hbase/lib/htrace-core4-4.2.0-incubating.jar:/opt/hbase/lib/httpclient-4.5.3.jar:/opt/hbase/lib/httpcore-4.4.6.jar:/opt/hbase/lib/jackson-annotations-2.9.0.jar:/opt/hbase/lib/jackson-core-2.9.2.jar:/opt/hbase/lib/jackson-core-asl-1.9.13.jar:/opt/hbase/lib/jackson-databind-2.9.2.jar:/opt/hbase/lib/jackson-jaxrs-1.8.3.jar:/opt/hbase/lib/jackson-jaxrs-base-2.9.2.jar:/opt/hbase/lib/jackson-jaxrs-json-provider-2.9.2.jar:/opt/hbase/lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/lib/jackson-module-jaxb-annotations-2.9.2.jar:/opt/hbase/lib/jackson-xc-1.8.3.jar:/opt/hbase/lib/jamon-runtime-2.4.1.jar:/opt/hbase/lib/java-xmlbuilder-0.4.jar:/opt/hbase/lib/javassist-3.20.0-GA.jar:/opt/hbase/lib/javax.annotation-api-1.2.jar:/opt/hbase/lib/javax.el-3.0.1-b08.jar:/opt/hbase/lib/javax.inject-2.5.0-b32.jar:/opt/hbase/lib/javax.servlet-api-3.1.0.jar:/opt/hbase/lib/javax.servlet.jsp-2.3.2.jar:/opt/hbase/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/hbase/lib/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/opt/hbase/lib/javax.servlet.jsp.jstl-1.2.2.jar:/opt/hbase/lib/javax.ws.rs-api-2.0.1.jar:/opt/hbase/lib/jaxb-api-2.2.12.jar:/opt/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/lib/jcodings-1.0.18.jar:/opt/hbase/lib/jersey-client-2.25.1.jar:/opt/hbase/lib/jersey-common-2.25.1.jar:/opt/hbase/lib/jersey-container-servlet-core-2.25.1.jar:/opt/hbase/lib/jersey-guava-2.25.1.jar:/opt/hbase/lib/jersey-media-jaxb-2.25.1.jar:/opt/hbase/lib/jersey-server-2.25.1.jar:/opt/hbase/lib/jets3t-0.9.0.jar:/opt/hbase/lib/jettison-1.3.8.jar:/opt/hbase/lib/jetty-6.1.26.jar:/opt/hbase/lib/jetty-http-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-io-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-jmx-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-jsp-9.2.19.v20160908.jar:/opt/hbase/lib/jetty-schemas-3.1.M0.jar:/opt/hbase/lib/jetty-security-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-server-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-servlet-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-sslengine-6.1.26.jar:/opt/hbase/lib/jetty-util-6.1.26.jar:/opt/hbase/lib/jetty-util-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-util-ajax-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-webapp-9.3.19.v20170502.jar:/opt/hbase/lib/jetty-xml-9.3.19.v20170502.jar:/opt/hbase/lib/joni-2.1.11.jar:/opt/hbase/lib/jsch-0.1.54.jar:/opt/hbase/lib/junit-4.12.jar:/opt/hbase/lib/leveldbjni-all-1.8.jar:/opt/hbase/lib/libthrift-0.9.3.jar:/opt/hbase/lib/log4j-1.2.17.jar:/opt/hbase/lib/metrics-core-3.2.1.jar:/opt/hbase/lib/netty-all-4.0.23.Final.jar:/opt/hbase/lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/opt/hbase/lib/osgi-resource-locator-1.0.1.jar:/opt/hbase/lib/paranamer-2.3.jar:/opt/hbase/lib/phoenix-5.0.0-HBase-2.0-client.jar:/opt/hbase/lib/phoenix-5.0.0-HBase-2.0-server.jar:/opt/hbase/lib/protobuf-java-2.5.0.jar:/opt/hbase/lib/remotecontent?filepath=com%2Fgoogle%2Fguava%2Fguava%2F13.0.1%2Fguava-13.0.1.jar:/opt/hbase/lib/slf4j-api-1.7.25.jar:/opt/hbase/lib/slf4j-log4j12-1.7.25.jar:/opt/hbase/lib/snappy-java-1.0.5.jar:/opt/hbase/lib/spymemcached-2.12.2.jar:/opt/hbase/lib/validation-api-1.1.0.Final.jar:/opt/hbase/lib/xmlenc-0.52.jar:/opt/hbase/lib/xz-1.0.jar:/opt/hbase/lib/zookeeper-3.4.10.jar::

 From what I can see, the the hbase jars are 2.0.2.

Francis

On 26/09/2018 1:04 AM, Jaanai Zhang wrote:
>
>
>     Is my method of installing HBase and Phoenix correct?
>
> Did you check versions of HBase that exists in your classpath?
>
>     Is this a compatibility issue with Guava?
>
> It isn't an exception which incompatible with Guava
>
> ----------------------------------------
>    Jaanai Zhang
>    Best regards!
>
>
>
> Francis Chuang <francischuang@apache.org 
> <ma...@apache.org>> 于2018年9月25日周二 下午8:25写道：
>
>     Thanks for taking a look, Jaanai!
>
>     Is my method of installing HBase and Phoenix correct? See
>     https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L12
>
>     Is this a compatibility issue with Guava?
>
>     Francis
>
>     On 25/09/2018 10:21 PM, Jaanai Zhang wrote:
>>
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>
>>     It looks like that HBase's Jars are incompatible.
>>
>>     ----------------------------------------
>>        Jaanai Zhang
>>        Best regards!
>>
>>
>>
>>     Francis Chuang <francischuang@apache.org
>>     <ma...@apache.org>> 于2018年9月25日周二 下午8:06写道：
>>
>>         Hi All,
>>
>>         I recently updated one of my Go apps to use Phoenix 5.0 with
>>         HBase
>>         2.0.2. I am using my Phoenix + HBase all in one docker image
>>         available
>>         here: https://github.com/Boostport/hbase-phoenix-all-in-one
>>
>>         This is the log/output from the exception:
>>
>>         RuntimeException: org.apache.phoenix.execute.CommitException:
>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>
>>         Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>         -> CommitException:
>>         org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>
>>         Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>         -> RetriesExhaustedWithDetailsException: Failed 1 action:
>>         org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>>
>>         Failed to build index for unexpected reason!
>>                      at
>>         org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>                      at
>>         org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>                      at
>>         org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>                      at
>>         org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>                      at
>>         org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>                      at
>>         org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>                  Caused by: java.lang.VerifyError: class
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1
>>         overrides
>>         final method
>>         compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>                      at java.lang.ClassLoader.defineClass1(Native Method)
>>                      at
>>         java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>                      at
>>         java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>                      at
>>         java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>                      at
>>         java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>                      at
>>         java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>                      at
>>         java.security.AccessController.doPrivileged(Native Method)
>>                      at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>                      at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>                      at
>>         java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>                      at
>>         org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>                      at
>>         org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>                      at
>>         org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>                      ... 18 more
>>                  : 1 time, servers with issues:
>>         9ac923bd5c9f,16020,1537875547341
>>
>>         I am not entirely sure if this is a bug, so have not opened a
>>         JIRA yet.
>>         Due to Guava issues with Tephra (see TEPHRA-181), I replaced
>>         HBase's
>>         Guava 11.0.2 with Guava 13.0.1. See
>>         https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>>
>>         Is Guava 13.0.1 compatible with this release (signaling that
>>         this might
>>         be cause by incompatible versions of Guava) or should I be
>>         investigating
>>         something else?
>>
>>         Francis
>>
>

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Jaanai Zhang <cl...@gmail.com>]

>
>
> Is my method of installing HBase and Phoenix correct?
>
Did you check versions of HBase that exists in your classpath?

Is this a compatibility issue with Guava?

It isn't an exception which incompatible with Guava

----------------------------------------
   Jaanai Zhang
   Best regards!



Francis Chuang <fr...@apache.org> 于2018年9月25日周二 下午8:25写道：

> Thanks for taking a look, Jaanai!
>
> Is my method of installing HBase and Phoenix correct? See
> https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L12
>
> Is this a compatibility issue with Guava?
>
> Francis
>
> On 25/09/2018 10:21 PM, Jaanai Zhang wrote:
>
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>> final method
>> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>              at java.lang.ClassLoader.defineClass1(Native Method)
>>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>              at
>>
>
> It looks like that HBase's Jars are incompatible.
>
> ----------------------------------------
>    Jaanai Zhang
>    Best regards!
>
>
>
> Francis Chuang <fr...@apache.org> 于2018年9月25日周二 下午8:06写道：
>
>> Hi All,
>>
>> I recently updated one of my Go apps to use Phoenix 5.0 with HBase
>> 2.0.2. I am using my Phoenix + HBase all in one docker image available
>> here: https://github.com/Boostport/hbase-phoenix-all-in-one
>>
>> This is the log/output from the exception:
>>
>> RuntimeException: org.apache.phoenix.execute.CommitException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action:
>> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>> Failed to build index for unexpected reason!
>>              at
>>
>> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>              at
>>
>> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>              at
>> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>          Caused by: java.lang.VerifyError: class
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>> final method
>> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>              at java.lang.ClassLoader.defineClass1(Native Method)
>>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>              at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>              at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>              at java.security.AccessController.doPrivileged(Native Method)
>>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>              at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>              at
>>
>> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>              at
>>
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>              ... 18 more
>>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
>> -> CommitException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action:
>> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>> Failed to build index for unexpected reason!
>>              at
>>
>> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>              at
>>
>> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>              at
>> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>          Caused by: java.lang.VerifyError: class
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>> final method
>> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>              at java.lang.ClassLoader.defineClass1(Native Method)
>>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>              at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>              at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>              at java.security.AccessController.doPrivileged(Native Method)
>>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>              at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>              at
>>
>> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>              at
>>
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>              ... 18 more
>>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
>> -> RetriesExhaustedWithDetailsException: Failed 1 action:
>> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>> Failed to build index for unexpected reason!
>>              at
>>
>> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>>              at
>>
>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>>              at
>>
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>>              at
>>
>> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>>              at
>> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>>              at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>>          Caused by: java.lang.VerifyError: class
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>> final method
>> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>>              at java.lang.ClassLoader.defineClass1(Native Method)
>>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>              at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>              at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>              at java.security.AccessController.doPrivileged(Native Method)
>>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>              at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>>              at
>>
>> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>>              at
>>
>> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>>              at
>>
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>>              at
>> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>>              ... 18 more
>>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
>>
>> I am not entirely sure if this is a bug, so have not opened a JIRA yet.
>> Due to Guava issues with Tephra (see TEPHRA-181), I replaced HBase's
>> Guava 11.0.2 with Guava 13.0.1. See
>>
>> https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>>
>> Is Guava 13.0.1 compatible with this release (signaling that this might
>> be cause by incompatible versions of Guava) or should I be investigating
>> something else?
>>
>> Francis
>>
>>
>

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Francis Chuang <fr...@apache.org>]

Thanks for taking a look, Jaanai!

Is my method of installing HBase and Phoenix correct? See 
https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L12

Is this a compatibility issue with Guava?

Francis

On 25/09/2018 10:21 PM, Jaanai Zhang wrote:
>
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>     final method
>     compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>                  at java.lang.ClassLoader.defineClass1(Native Method)
>                  at
>     java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>                  at
>
> It looks like that HBase's Jars are incompatible.
>
> ----------------------------------------
>    Jaanai Zhang
>    Best regards!
>
>
>
> Francis Chuang <francischuang@apache.org 
> <ma...@apache.org>> 于2018年9月25日周二 下午8:06写道：
>
>     Hi All,
>
>     I recently updated one of my Go apps to use Phoenix 5.0 with HBase
>     2.0.2. I am using my Phoenix + HBase all in one docker image
>     available
>     here: https://github.com/Boostport/hbase-phoenix-all-in-one
>
>     This is the log/output from the exception:
>
>     RuntimeException: org.apache.phoenix.execute.CommitException:
>     org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>     Failed 1 action:
>     org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>     Failed to build index for unexpected reason!
>                  at
>     org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>                  at
>     org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>                  at
>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>              Caused by: java.lang.VerifyError: class
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>     final method
>     compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>                  at java.lang.ClassLoader.defineClass1(Native Method)
>                  at
>     java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>                  at
>     java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>                  at
>     java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>                  at
>     java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>                  at java.security.AccessController.doPrivileged(Native
>     Method)
>                  at
>     java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>                  at
>     sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>                  at
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>                  at
>     org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>                  at
>     org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>                  at
>     org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>                  ... 18 more
>              : 1 time, servers with issues:
>     9ac923bd5c9f,16020,1537875547341
>     -> CommitException:
>     org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>     Failed 1 action:
>     org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>     Failed to build index for unexpected reason!
>                  at
>     org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>                  at
>     org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>                  at
>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>              Caused by: java.lang.VerifyError: class
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>     final method
>     compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>                  at java.lang.ClassLoader.defineClass1(Native Method)
>                  at
>     java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>                  at
>     java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>                  at
>     java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>                  at
>     java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>                  at java.security.AccessController.doPrivileged(Native
>     Method)
>                  at
>     java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>                  at
>     sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>                  at
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>                  at
>     org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>                  at
>     org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>                  at
>     org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>                  ... 18 more
>              : 1 time, servers with issues:
>     9ac923bd5c9f,16020,1537875547341
>     -> RetriesExhaustedWithDetailsException: Failed 1 action:
>     org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
>     Failed to build index for unexpected reason!
>                  at
>     org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>                  at
>     org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>                  at
>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>                  at
>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>                  at
>     org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>                  at
>     org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>                  at
>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>                  at
>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>              Caused by: java.lang.VerifyError: class
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
>     final method
>     compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>                  at java.lang.ClassLoader.defineClass1(Native Method)
>                  at
>     java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>                  at
>     java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>                  at
>     java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>                  at
>     java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>                  at java.security.AccessController.doPrivileged(Native
>     Method)
>                  at
>     java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>                  at
>     sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>                  at
>     org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>                  at
>     org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>                  at
>     org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>                  at
>     org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>                  at
>     org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>                  ... 18 more
>              : 1 time, servers with issues:
>     9ac923bd5c9f,16020,1537875547341
>
>     I am not entirely sure if this is a bug, so have not opened a JIRA
>     yet.
>     Due to Guava issues with Tephra (see TEPHRA-181), I replaced HBase's
>     Guava 11.0.2 with Guava 13.0.1. See
>     https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>
>     Is Guava 13.0.1 compatible with this release (signaling that this
>     might
>     be cause by incompatible versions of Guava) or should I be
>     investigating
>     something else?
>
>     Francis
>

Re: Phoenix 5.0 could not commit transaction: org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason!

[Jaanai Zhang <cl...@gmail.com>]

>
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
> final method
> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>              at java.lang.ClassLoader.defineClass1(Native Method)
>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>              at
>

It looks like that HBase's Jars are incompatible.

----------------------------------------
   Jaanai Zhang
   Best regards!



Francis Chuang <fr...@apache.org> 于2018年9月25日周二 下午8:06写道：

> Hi All,
>
> I recently updated one of my Go apps to use Phoenix 5.0 with HBase
> 2.0.2. I am using my Phoenix + HBase all in one docker image available
> here: https://github.com/Boostport/hbase-phoenix-all-in-one
>
> This is the log/output from the exception:
>
> RuntimeException: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action:
> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
> Failed to build index for unexpected reason!
>              at
>
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>              at
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>              at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>              at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>          Caused by: java.lang.VerifyError: class
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
> final method
> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>              at java.lang.ClassLoader.defineClass1(Native Method)
>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>              at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>              at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>              at java.security.AccessController.doPrivileged(Native Method)
>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>              at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>              at
>
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>              at
>
> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>              at
>
> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>              at
>
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>              at
>
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>              ... 18 more
>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
> -> CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action:
> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
> Failed to build index for unexpected reason!
>              at
>
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>              at
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>              at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>              at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>          Caused by: java.lang.VerifyError: class
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
> final method
> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>              at java.lang.ClassLoader.defineClass1(Native Method)
>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>              at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>              at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>              at java.security.AccessController.doPrivileged(Native Method)
>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>              at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>              at
>
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>              at
>
> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>              at
>
> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>              at
>
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>              at
>
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>              ... 18 more
>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
> -> RetriesExhaustedWithDetailsException: Failed 1 action:
> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException:
> Failed to build index for unexpected reason!
>              at
>
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
>              at
>
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
>              at
>
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487)
>              at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854)
>              at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
>              at
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
>              at
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
>              at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
>              at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>              at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>          Caused by: java.lang.VerifyError: class
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides
> final method
> compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I
>              at java.lang.ClassLoader.defineClass1(Native Method)
>              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>              at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>              at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>              at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>              at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>              at java.security.AccessController.doPrivileged(Native Method)
>              at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>              at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>              at
>
> org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82)
>              at
>
> org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57)
>              at
>
> org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52)
>              at
>
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90)
>              at
>
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503)
>              at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
>              ... 18 more
>          : 1 time, servers with issues: 9ac923bd5c9f,16020,1537875547341
>
> I am not entirely sure if this is a bug, so have not opened a JIRA yet.
> Due to Guava issues with Tephra (see TEPHRA-181), I replaced HBase's
> Guava 11.0.2 with Guava 13.0.1. See
>
> https://github.com/Boostport/hbase-phoenix-all-in-one/blob/master/Dockerfile#L43
>
> Is Guava 13.0.1 compatible with this release (signaling that this might
> be cause by incompatible versions of Guava) or should I be investigating
> something else?
>
> Francis
>
>