You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Sergey Soldatov (JIRA)" <ji...@apache.org> on 2017/08/08 21:44:00 UTC
[jira] [Commented] (PHOENIX-4075) Executor already shutdown
building secondary index
[ https://issues.apache.org/jira/browse/PHOENIX-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119077#comment-16119077 ]
Sergey Soldatov commented on PHOENIX-4075:
------------------------------------------
That's may be a side effect of PHOENIX-3360 and follows jiras. To properly set the configuration you need to have a separate copy of hbase-site.xml for PQS with removed controllerfactory property.
> Executor already shutdown building secondary index
> --------------------------------------------------
>
> Key: PHOENIX-4075
> URL: https://issues.apache.org/jira/browse/PHOENIX-4075
> Project: Phoenix
> Issue Type: Bug
> Environment: 3 Node HBase cluster running version 1.2.0-cdh5.7.2
> Phoenix version 4.7.0 using thin client + query server
> Reporter: Brian Thomas
>
> I started to experience an issue trying to insert some data into our HBase cluster using phoenix. This seems to have start occurring after I added a secondary index on one of my columns.
> The issue seems to resolve temporarily with an HBase restart. However, it eventually falls back into this state and once it gets into this state, it's not possible to insert data anymore. When my application is running, it tries to insert 80k rows and then issues a single commit. However, the batch size doesn't seem to affect it as I cannot even insert a single row when it gets into this state. The full stack trace I see is
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 131563 actions: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexp
> ected reason!
> at org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:180)
> at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:205)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3088)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2817)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:751)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:713)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2142)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.RejectedExecutionException: Executor already shutdown
> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.startTask(MoreExecutors.java:327)
> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:251)
> at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
> at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:58)
> at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:143)
> at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:273)
> at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:202)
> ... 17 more
> : 131563 times,
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
> at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1663)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:982)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:996)
> at org.apache.phoenix.execute.MutationState.send(MutationState.java:951)
> ... 22 more
> at org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2315)
> at org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:46)
> at org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:107)
> at org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:377)
> at org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:375)
> at org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:666)
> at org.apache.calcite.avatica.remote.RemoteMeta.commit(RemoteMeta.java:375)
> at org.apache.calcite.avatica.AvaticaConnection.commit(AvaticaConnection.java:175)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)