You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/23 18:13:42 UTC
[GitHub] [pinot] suddendust opened a new issue #7476: Pinot Ingestion Rate Sudden Drop to 0
suddendust opened a new issue #7476:
URL: https://github.com/apache/pinot/issues/7476
We ingested data from Kafka into six Pinot real-time tables. A few hours back, the ingestion on all the tables dropped to 0. Server logs:
```
expected=ONLINE, found=OFFLINE, creationTime=1632409068678, modifiedTime=1632409072718, version=13, waitingFor=CurrentStateMatch, resource=spanEventView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:15.903 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:25.914 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409104146, version=16, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:35.953 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:45.969 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:55.977 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:05.987 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:15.995 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:26.019 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:36.027 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:46.040 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:56.050 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:06.061 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
Exception in thread "rawTraceView__0__1455__20210923T1346Z" Exception in thread "rawServiceView__0__3350__20210923T1224Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:57)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:521)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.core.realtime.impl.dictionary.BytesOffHeapMutableDictionary.getMinVal(BytesOffHeapMutableDictionary.java:115)
at org.apache.pinot.core.realtime.impl.dictionary.BytesOffHeapMutableDictionary.getMinVal(BytesOffHeapMutableDictionary.java:34)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:525)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "backendEntityView__2__23061__20210923T1352Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:663)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:475)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "serviceCallView__2__9949__20210923T1352Z" Exception in thread "spanEventView__5__869__20210923T1454Z" Exception in thread "spanEventView__1__33514__20210923T1454Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1845)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3773)
at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:195)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:469)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getPropertySafe(ScriptBytecodeAdapter.java:477)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getPropertySpreadSafe(ScriptBytecodeAdapter.java:485)
at Script1.run(Script1.groovy:1)
at org.apache.pinot.core.data.function.GroovyFunctionEvaluator.evaluate(GroovyFunctionEvaluator.java:95)
at org.apache.pinot.core.data.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:95)
at org.apache.pinot.core.data.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:82)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:491)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at java.util.HashMap.putVal(HashMap.java:642)
at java.util.HashMap.put(HashMap.java:612)
at org.apache.pinot.spi.data.readers.GenericRow.putValue(GenericRow.java:127)
at org.apache.pinot.plugin.inputformat.avro.AvroRecordExtractor.extract(AvroRecordExtractor.java:70)
at org.apache.pinot.plugin.inputformat.avro.AvroRecordExtractor.extract(AvroRecordExtractor.java:38)
at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:110)
at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:115)
at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:51)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:471)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.core.realtime.impl.dictionary.BaseOffHeapMutableDictionary.indexValue(BaseOffHeapMutableDictionary.java:361)
at org.apache.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:61)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:521)
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
at java.lang.Thread.run(Thread.java:748)
2021/09/23 15:00:16.067 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409215577, version=11, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:26.077 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:36.093 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:46.103 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:56.113 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409255125, version=22, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:01:06.122 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23040__20210923T0619Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409266062, version=48, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:01:16.215 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 190391ms
2021/09/23 15:01:16.222 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Pinot server ready
2021/09/23 15:30:06.057 WARN [ZkBaseDataAccessor] [ZkClient-EventThread-23-apache-pinot-zookeeper-bitnami-headless.pinot.svc.cluster.local:2181] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/74dc1531-5db0-47a4-b97c-95cabc4067dc=-101}
```
Controller logs:
```
Cluster: prod-green
2021/09/23 14:01:12.778 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a392b5d6_DEFAULT)] Event a392b5d6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:12.879 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c9cc5f8e_DEFAULT)] Event c9cc5f8e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:12.996 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fb813904_DEFAULT)] Event fb813904_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.184 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event d0ca4d41_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:01:13.200 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d0ca4d41_DEFAULT)] Event d0ca4d41_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.325 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c9563711_DEFAULT)] Event c9563711_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.405 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c7d1eaa6_DEFAULT)] Event c7d1eaa6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.798 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(43b22c37_DEFAULT)] Event 43b22c37_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.001 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3f59c606_DEFAULT)] Event 3f59c606_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.089 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ebacdc57_DEFAULT)] Event ebacdc57_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.276 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6743a09e_DEFAULT)] Event 6743a09e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.004 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f6334c16_DEFAULT)] Event f6334c16_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.117 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(74e299c8_DEFAULT)] Event 74e299c8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.272 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8d39b730_DEFAULT)] Event 8d39b730_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:20.303 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(bf2e9df7_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/2d3beedd-4731-4036-a8d9-20bbc7b27d27=-101}
2021/09/23 14:01:20.304 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(bf2e9df7_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/2d3beedd-4731-4036-a8d9-20bbc7b27d27=-101}
2021/09/23 14:01:20.376 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bf2e9df7_DEFAULT)] Event bf2e9df7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:20.497 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(c86a979c_DEFAULT)] Event c86a979c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:01:27.242 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(34702a6e_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/99bdeb71-78b9-41da-b9ca-9c9acf6f18db=-101}
2021/09/23 14:01:27.243 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(34702a6e_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/99bdeb71-78b9-41da-b9ca-9c9acf6f18db=-101}
2021/09/23 14:02:19.624 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 597a12e9_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:03:20.125 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9313c732_DEFAULT)] Event 9313c732_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:03:38.595 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632405818595 start=1632405563326
2021/09/23 14:03:39.761 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(1c6e25a9_DEFAULT)] Event 1c6e25a9_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:03:41.917 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632405821917 start=1632405559500
2021/09/23 14:03:52.468 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event c4136a1e_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:32.970 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632405872970 start=1632405653317
2021/09/23 14:04:33.808 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632405873808 start=1632405653837
2021/09/23 14:04:39.609 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f2d45c6d_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:39.993 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f7894bf7_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:49.945 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2152fb5b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:05:57.279 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1248c422_DEFAULT)] Event 1248c422_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.369 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fd9b0a0a_DEFAULT)] Event fd9b0a0a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.488 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(20a00442_DEFAULT)] Event 20a00442_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.634 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4424c7c6_DEFAULT)] Event 4424c7c6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.685 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 6ac40d3f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:05:57.733 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6ac40d3f_DEFAULT)] Event 6ac40d3f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.824 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3fc8e670_DEFAULT)] Event 3fc8e670_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.189 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b4358f84_DEFAULT)] Event b4358f84_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.353 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a145559b_DEFAULT)] Event a145559b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.455 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8c54bc97_DEFAULT)] Event 8c54bc97_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.036 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6c2378d1_DEFAULT)] Event 6c2378d1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.135 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2af892ef_DEFAULT)] Event 2af892ef_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.276 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(159f3e0a_DEFAULT)] Event 159f3e0a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.370 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(99da1fd0_DEFAULT)] Event 99da1fd0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.472 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a9e2c61a_DEFAULT)] Event a9e2c61a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.603 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(58858c7a_DEFAULT)] Event 58858c7a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.719 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b12a1c43_DEFAULT)] Event b12a1c43_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.843 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9acf27ca_DEFAULT)] Event 9acf27ca_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.404 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ea5fb5a1_DEFAULT)] Event ea5fb5a1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.525 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9a4328a6_DEFAULT)] Event 9a4328a6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.609 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e90911a7_DEFAULT)] Event e90911a7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:14.619 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8f49d193_DEFAULT)] Event 8f49d193_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:06:16.217 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event e399e4d8_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:06:27.437 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a66c4822_DEFAULT)] Event a66c4822_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:06:39.954 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8152f13b_DEFAULT)] Event 8152f13b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:07:46.932 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632406066932 start=1632405877806
2021/09/23 14:07:47.786 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632406067786 start=1632405877806
2021/09/23 14:08:29.048 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406109048 start=1632405824922
2021/09/23 14:08:30.287 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406110286 start=1632405821613
2021/09/23 14:09:00.486 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(b49b11d8_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/dfdafac4-90d8-487c-909a-17cb69d25100=-101}
2021/09/23 14:09:00.487 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(b49b11d8_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/dfdafac4-90d8-487c-909a-17cb69d25100=-101}
2021/09/23 14:09:00.765 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b1e6c730_DEFAULT)] Event b1e6c730_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:06.571 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2fc509c5_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:09:07.656 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4dedf5c4_DEFAULT)] Event 4dedf5c4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:11.922 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4dac10bd_DEFAULT)] Event 4dac10bd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:14.788 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 8091ca04_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:09.733 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__2__33540__20210923T1342Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.736 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__6__860__20210923T1346Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.736 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__6__861__20210923T1349Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.737 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 3 segments with no online replicas
2021/09/23 14:10:09.737 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:10:31.380 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2f10fe65_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8c9bacc8-fb8b-4f3c-8733-3576f80406f3=-101}
2021/09/23 14:10:31.382 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2f10fe65_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-2.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/75beafe3-955d-4a07-86ad-aaae0d8a694d=-101}
2021/09/23 14:10:31.529 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(10ad37c3_DEFAULT)] Event 10ad37c3_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:10:41.024 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 87097d51_TASK(LiveInstanceChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:41.345 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event ff683f83_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:41.725 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d543cb23_DEFAULT)] Event d543cb23_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:41.829 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(830983b5_DEFAULT)] Event 830983b5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:41.916 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(66119eb0_DEFAULT)] Event 66119eb0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.135 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b18a3b8c_DEFAULT)] Event b18a3b8c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.267 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bb1f2b85_DEFAULT)] Event bb1f2b85_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.359 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f315c94d_DEFAULT)] Event f315c94d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.451 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9ffd9f20_DEFAULT)] Event 9ffd9f20_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.550 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(17d66edd_DEFAULT)] Event 17d66edd_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:11:15.854 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632406275854 start=1632406071930
2021/09/23 14:11:19.585 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632406279585 start=1632406069950
2021/09/23 14:12:07.724 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(e14c411e_DEFAULT)] Event e14c411e_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:12:21.039 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(338cf08d_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8a810e52-db6a-4e5e-98a8-04e1036c6658=-101}
2021/09/23 14:12:21.040 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(338cf08d_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8a810e52-db6a-4e5e-98a8-04e1036c6658=-101}
2021/09/23 14:12:21.295 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9e0f9405_DEFAULT)] Event 9e0f9405_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:13:14.094 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406394093 start=1632406113386
2021/09/23 14:13:33.649 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(e89fc9ed_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-1.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d116a355-20d9-4a6d-9603-91a4c3e4f94e=-101}
2021/09/23 14:13:33.650 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(e89fc9ed_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-1.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d116a355-20d9-4a6d-9603-91a4c3e4f94e=-101}
2021/09/23 14:13:33.820 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ffb29267_DEFAULT)] Event ffb29267_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:13:41.159 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406421159 start=1632406112066
2021/09/23 14:14:57.334 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d25f8721_DEFAULT)] Event d25f8721_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.456 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d68becb0_DEFAULT)] Event d68becb0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.551 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d67c676d_DEFAULT)] Event d67c676d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.786 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(92a57d49_DEFAULT)] Event 92a57d49_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.949 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f24f8a65_DEFAULT)] Event f24f8a65_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:58.057 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4a5b10a5_DEFAULT)] Event 4a5b10a5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.264 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(66d8b7cd_DEFAULT)] Event 66d8b7cd_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.390 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(caf4e251_DEFAULT)] Event caf4e251_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.530 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8d53b02d_DEFAULT)] Event 8d53b02d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.662 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(957556b3_DEFAULT)] Event 957556b3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.817 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0576c0fe_DEFAULT)] Event 0576c0fe_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.982 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f4fa5ee7_DEFAULT)] Event f4fa5ee7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:01.070 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8ab8af28_DEFAULT)] Event 8ab8af28_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:09.804 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__5__859__20210923T1314Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:15:09.806 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 1 segments with no online replicas
2021/09/23 14:15:09.806 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:15:19.682 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(04980f3b_DEFAULT)] Event 04980f3b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:15:22.552 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(19589fce_DEFAULT)] Event 19589fce_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:17:18.048 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406638048 start=1632406397119
2021/09/23 14:17:19.030 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406639029 start=1632406424164
2021/09/23 14:18:38.995 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a26e9f71_DEFAULT)] Event a26e9f71_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:18:43.310 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(5769fcf5_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/837e0cf5-ae48-44aa-b4c8-5ae39319b266=-101}
2021/09/23 14:18:46.222 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b6a2f71f_DEFAULT)] Event b6a2f71f_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:19:02.897 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ed0d0b7c_DEFAULT)] Event ed0d0b7c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.009 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a99f1bc5_DEFAULT)] Event a99f1bc5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.103 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9beb2d6c_DEFAULT)] Event 9beb2d6c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.212 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(14e0215e_DEFAULT)] Event 14e0215e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.322 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(789bc57a_DEFAULT)] Event 789bc57a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.671 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(16f58f4c_DEFAULT)] Event 16f58f4c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.803 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b3f5e4ca_DEFAULT)] Event b3f5e4ca_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.902 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8542debc_DEFAULT)] Event 8542debc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.933 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 29695a3d_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:04.002 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(29695a3d_DEFAULT)] Event 29695a3d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.324 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c5c89c4d_DEFAULT)] Event c5c89c4d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.435 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d60cea69_DEFAULT)] Event d60cea69_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.565 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d9a446e3_DEFAULT)] Event d9a446e3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.573 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 89c8b6e3_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:05.668 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(5415e7a7_DEFAULT)] Event 5415e7a7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.793 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(923d18f5_DEFAULT)] Event 923d18f5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.916 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4f4c0076_DEFAULT)] Event 4f4c0076_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.026 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2a060e26_DEFAULT)] Event 2a060e26_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.137 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4d9f0638_DEFAULT)] Event 4d9f0638_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.261 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f37a8dd0_DEFAULT)] Event f37a8dd0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.365 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(77936c0c_DEFAULT)] Event 77936c0c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.545 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7c199f34_DEFAULT)] Event 7c199f34_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.637 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0642ba83_DEFAULT)] Event 0642ba83_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.778 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fef10f2f_DEFAULT)] Event fef10f2f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.877 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(725dca81_DEFAULT)] Event 725dca81_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.064 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c940957d_DEFAULT)] Event c940957d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.115 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event a35f028c_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:07.153 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a35f028c_DEFAULT)] Event a35f028c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.401 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f925dd8_DEFAULT)] Event 7f925dd8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.533 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a995bf0b_DEFAULT)] Event a995bf0b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.675 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(90adb41d_DEFAULT)] Event 90adb41d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:21:48.353 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(f99fc6a1_DEFAULT)] Event f99fc6a1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:21:48.459 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 00eedc5e_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:22:17.789 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(0682eb12_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d720424e-9252-4dd9-a2a0-ae14025d1b34=-101}
2021/09/23 14:22:18.077 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(406d020d_DEFAULT)] Event 406d020d_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:24:17.058 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b157060e_DEFAULT)] Event b157060e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.201 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(14bc7478_DEFAULT)] Event 14bc7478_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.282 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c88f2947_DEFAULT)] Event c88f2947_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.927 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1466657d_DEFAULT)] Event 1466657d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.034 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(96009226_DEFAULT)] Event 96009226_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.131 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f63f7c1_DEFAULT)] Event 7f63f7c1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.243 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(543c7fdf_DEFAULT)] Event 543c7fdf_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.330 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(192ff523_DEFAULT)] Event 192ff523_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.201 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ec890f2e_DEFAULT)] Event ec890f2e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.318 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7a44ddd3_DEFAULT)] Event 7a44ddd3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.410 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0e70de49_DEFAULT)] Event 0e70de49_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.558 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(56ddae42_DEFAULT)] Event 56ddae42_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.671 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9cbf5245_DEFAULT)] Event 9cbf5245_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.801 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(92890cc6_DEFAULT)] Event 92890cc6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.900 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fcc6929d_DEFAULT)] Event fcc6929d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.011 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3511a196_DEFAULT)] Event 3511a196_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.152 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2b6be989_DEFAULT)] Event 2b6be989_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.302 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d8d22884_DEFAULT)] Event d8d22884_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.396 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(168eb610_DEFAULT)] Event 168eb610_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.496 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(77ba9443_DEFAULT)] Event 77ba9443_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.625 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3760a7e7_DEFAULT)] Event 3760a7e7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.743 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a18d1f06_DEFAULT)] Event a18d1f06_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.842 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3632bcb8_DEFAULT)] Event 3632bcb8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.940 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2d8c8414_DEFAULT)] Event 2d8c8414_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.033 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7021fcc9_DEFAULT)] Event 7021fcc9_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.170 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8499a211_DEFAULT)] Event 8499a211_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.258 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(26eb1da1_DEFAULT)] Event 26eb1da1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.372 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ba1c9c9d_DEFAULT)] Event ba1c9c9d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:25:08.578 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b28448e5_DEFAULT)] Event b28448e5_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 2 segments with no online replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.940 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 10 segments with no online replicas
2021/09/23 14:25:09.940 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:50.994 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 171a1662_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:26:13.785 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2ddb8cff_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:26:16.433 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 7ca4b237_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:28:05.170 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407285169 start=1632407036623
2021/09/23 14:29:41.633 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 5a420789_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment rawTraceView__2__1446__20210923T1127Z of table rawTraceView_REALTIME has no online replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Table rawTraceView_REALTIME has 2 segments with no online replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:30:09.952 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__0__9856__20210923T1319Z of table serviceCallView_REALTIME has no online replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__3__9908__20210923T1304Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Table serviceCallView_REALTIME has 5 segments with no online replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.009 WARN [SegmentStatusChecker] [pool-9-thread-3] Table spanEventView_REALTIME has 28 segments with no online replicas
2021/09/23 14:30:10.009 WARN [SegmentStatusChecker] [pool-9-thread-3] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:31:53.077 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2c52bb4c_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:31:54.082 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(075b0909_DEFAULT)] Event 075b0909_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.104 WARN [SegmentCompletionFSM_spanEventView__4__877__20210923T1421Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2195000000 now=1632407514104 start=1632407285691
2021/09/23 14:31:54.179 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ea83ad38_DEFAULT)] Event ea83ad38_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2c918baf_DEFAULT)] Event 2c918baf_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.450 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(15865d09_DEFAULT)] Event 15865d09_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.566 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d4673f42_DEFAULT)] Event d4673f42_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.618 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407514618 start=1632407288189
2021/09/23 14:31:54.664 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bb25679c_DEFAULT)] Event bb25679c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.980 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3bae3e48_DEFAULT)] Event 3bae3e48_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.110 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b2e60d7f_DEFAULT)] Event b2e60d7f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bad8d025_DEFAULT)] Event bad8d025_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.374 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3a8f1e4d_DEFAULT)] Event 3a8f1e4d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.498 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ac0a67aa_DEFAULT)] Event ac0a67aa_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.630 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4121bd6a_DEFAULT)] Event 4121bd6a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.815 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a55e8cef_DEFAULT)] Event a55e8cef_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.951 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(aef2a821_DEFAULT)] Event aef2a821_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.039 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(526c2e71_DEFAULT)] Event 526c2e71_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.138 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(253710c3_DEFAULT)] Event 253710c3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.352 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d5946f1b_DEFAULT)] Event d5946f1b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.507 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6b211404_DEFAULT)] Event 6b211404_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.628 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d93767b8_DEFAULT)] Event d93767b8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.708 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0a8a7d92_DEFAULT)] Event 0a8a7d92_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.509 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1fc64af6_DEFAULT)] Event 1fc64af6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.696 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1ac7573a_DEFAULT)] Event 1ac7573a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.776 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(949e7238_DEFAULT)] Event 949e7238_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:32:04.598 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407524598 start=1632407277460
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:33:26.019 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 58778b0f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:33:27.831 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 39460e63_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:11.070 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 57bff217_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:13.201 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 81f18031_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:13.489 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 9d1ecc43_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:14.463 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 155dd94f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:35:10.013 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:35:10.014 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 14:35:10.014 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__3__9908__20210923T1304Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__3__9909__20210923T1346Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 4 segments with no online replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.073 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 20 segments with no online replicas
2021/09/23 14:35:10.073 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.948 WARN [SegmentCompletionFSM_spanEventView__4__877__20210923T1421Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2195000000 now=1632407710948 start=1632407517123
2021/09/23 14:35:12.238 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407712238 start=1632407521300
2021/09/23 14:36:24.736 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407784735 start=1632407527616
2021/09/23 14:37:21.728 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 052d00c2_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:12.172 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 543abb06_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:18.941 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632407898941 start=1632406279036
2021/09/23 14:38:19.597 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1a4b4d7e_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:20.664 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1740608b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:25.173 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632407905173 start=1632406863796
2021/09/23 14:39:24.782 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bcc2829e_DEFAULT)] Event bcc2829e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:24.888 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(383e5361_DEFAULT)] Event 383e5361_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:24.975 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(97671be4_DEFAULT)] Event 97671be4_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.243 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(24f2e692_DEFAULT)] Event 24f2e692_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.447 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7a461e88_DEFAULT)] Event 7a461e88_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.606 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(51f01163_DEFAULT)] Event 51f01163_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.696 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(639db461_DEFAULT)] Event 639db461_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.801 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b5ff5629_DEFAULT)] Event b5ff5629_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.945 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c5cf649f_DEFAULT)] Event c5cf649f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.036 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ae150bbc_DEFAULT)] Event ae150bbc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.165 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(732ead7d_DEFAULT)] Event 732ead7d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8b5e4d6c_DEFAULT)] Event 8b5e4d6c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:34.983 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 0038b802_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:39:36.839 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407976839 start=1632407791886
2021/09/23 14:39:37.239 WARN [SegmentCompletionFSM_spanEventView__7__884__20210923T1425Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2212500000 now=1632407977239 start=1632407792258
2021/09/23 14:39:38.457 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(c4518cce_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/84a0363a-63a1-4971-a696-da778ff71662=-101}
2021/09/23 14:39:38.619 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(929327fc_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/76229ce7-2ac6-4ad3-9845-c5b11824381a=-101}
2021/09/23 14:39:43.413 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 45c9393a_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:41:16.409 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(5e3b47bf_DEFAULT)] Event 5e3b47bf_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:17.465 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(f218dbcd_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/641328b3-20ac-4c25-9286-c9dc7826f5e8=-101}
2021/09/23 14:41:17.706 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(56482011_DEFAULT)] Event 56482011_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:19.036 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d7aac4f0_DEFAULT)] Event d7aac4f0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:19.313 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 25783ff5_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:41:19.915 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(41561440_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/cc658de2-0e49-4166-bb30-7d76e089c0ec=-101}
2021/09/23 14:41:19.916 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(41561440_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/cc658de2-0e49-4166-bb30-7d76e089c0ec=-101}
2021/09/23 14:41:20.077 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(106686a3_DEFAULT)] Event 106686a3_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:35.098 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632408095098 start=1632406641538
2021/09/23 14:41:36.682 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632408096682 start=1632406642035
2021/09/23 14:43:26.799 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4752001c_DEFAULT)] Event 4752001c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:43:28.394 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2b064936_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/49d8cdcb-c396-464b-9708-32bb0663b8a5=-101}
2021/09/23 14:43:28.577 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(92bf9b02_DEFAULT)] Event 92bf9b02_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:33.659 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(f32c6d76_DEFAULT)] Event f32c6d76_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:39.457 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event d08f735b_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:44:39.657 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(7678d829_DEFAULT)] Event 7678d829_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:59.208 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(69758db7_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f2d95b25-48fe-4138-980a-5643f6adc295=-101}
2021/09/23 14:44:59.208 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(69758db7_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f2d95b25-48fe-4138-980a-5643f6adc295=-101}
2021/09/23 14:44:59.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b46a1e3b_DEFAULT)] Event b46a1e3b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:59.688 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event b638c3e6_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:45:01.507 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(72516529_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/4718ffa9-8583-43ea-ba6e-abe8e0f1f505=-101}
2021/09/23 14:45:01.678 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(97af3835_DEFAULT)] Event 97af3835_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:14.547 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(2b6c7a15_DEFAULT)] Event 2b6c7a15_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:19.690 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2b3de2ae_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-2.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d9ace221-bfed-4ba0-bdc5-7b9885317fcc=-101, /pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/aa3390e1-2407-4bf5-bbd7-b0fcb8e09a09=-101}
2021/09/23 14:45:19.691 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2b3de2ae_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/aa3390e1-2407-4bf5-bbd7-b0fcb8e09a09=-101}
2021/09/23 14:45:19.904 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ff9ed343_DEFAULT)] Event ff9ed343_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:33.941 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(58ac53c8_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/c175ccea-2fcd-46d8-81c9-844756319079=-101}
2021/09/23 14:45:34.092 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(02795e6c_DEFAULT)] Event 02795e6c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:05.103 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d38e1b94_DEFAULT)] Event d38e1b94_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:11.107 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(deea182d_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/4cfca853-a822-4d8c-baca-8fcdc6a9244c=-101}
2021/09/23 14:47:11.360 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8243fec6_DEFAULT)] Event 8243fec6_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:15.547 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(561261bd_DEFAULT)] Event 561261bd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:41.292 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9bdacc59_DEFAULT)] Event 9bdacc59_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:50.375 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9a36841c_DEFAULT)] Event 9a36841c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:56.034 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(6072a4b0_DEFAULT)] Event 6072a4b0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:48:34.470 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(c4bd1e84_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/414d6a14-404b-4cdb-a2b9-df386b0cc66f=-101}
2021/09/23 14:48:34.657 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(68708ceb_DEFAULT)] Event 68708ceb_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:48:45.468 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(633b5f48_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/88020225-a093-4af5-b003-f96bd85e0068=-101}
2021/09/23 14:48:45.469 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(633b5f48_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/88020225-a093-4af5-b003-f96bd85e0068=-101}
2021/09/23 14:48:45.727 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9e95b70d_DEFAULT)] Event 9e95b70d_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:50:33.714 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(dc0f2b34_DEFAULT)] Event dc0f2b34_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:50:40.483 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(1267f641_DEFAULT)] Event 1267f641_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:51:01.340 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(0ca9415a_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/772f26b3-0e8b-4d2b-989b-adc816280ff1=-101}
2021/09/23 14:51:01.606 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(3e9132c4_DEFAULT)] Event 3e9132c4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:51:01.701 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 9df6b091_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:51:08.222 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(e3dc68f4_DEFAULT)] Event e3dc68f4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:52:04.277 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a3f20ba0_DEFAULT)] Event a3f20ba0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:52:26.758 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(c354ee27_DEFAULT)] Event c354ee27_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:53:29.297 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(468f4ed5_DEFAULT)] Event 468f4ed5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.432 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fd296a0d_DEFAULT)] Event fd296a0d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.548 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f094c70_DEFAULT)] Event 7f094c70_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.697 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(20e151db_DEFAULT)] Event 20e151db_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.852 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c426c27c_DEFAULT)] Event c426c27c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.971 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4b61c912_DEFAULT)] Event 4b61c912_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.104 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(16257a88_DEFAULT)] Event 16257a88_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.216 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(5cef74b5_DEFAULT)] Event 5cef74b5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.379 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e205097e_DEFAULT)] Event e205097e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.481 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9a283bd5_DEFAULT)] Event 9a283bd5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.593 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ab809be6_DEFAULT)] Event ab809be6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:54:19.371 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(24844f37_DEFAULT)] Event 24844f37_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:54:28.693 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(e869bd7b_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/668520b0-95db-45a3-abf5-178dc4b5a4a5=-101}
2021/09/23 14:54:28.694 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(e869bd7b_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/668520b0-95db-45a3-abf5-178dc4b5a4a5=-101}
2021/09/23 14:54:28.939 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d65706a4_DEFAULT)] Event d65706a4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:55:35.648 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632408935648 start=1632408760513
2021/09/23 14:55:38.305 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632408938305 start=1632408754425
2021/09/23 14:55:44.644 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(60a61425_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/002aa711-82f9-4bf7-86be-1ced5c93d47c=-101}
2021/09/23 14:55:44.866 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(382c5177_DEFAULT)] Event 382c5177_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:56:32.517 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(dfecd0d6_DEFAULT)] Event dfecd0d6_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:56:32.734 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f8ad9360_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:56:58.690 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(84422a6d_DEFAULT)] Event 84422a6d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:58.794 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3b450013_DEFAULT)] Event 3b450013_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:58.931 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(07f2036b_DEFAULT)] Event 07f2036b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.054 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(595d86e1_DEFAULT)] Event 595d86e1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.154 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b16dc905_DEFAULT)] Event b16dc905_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.291 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c7c94c77_DEFAULT)] Event c7c94c77_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.386 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(594c8adc_DEFAULT)] Event 594c8adc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.568 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(104453ae_DEFAULT)] Event 104453ae_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.658 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(84d42d93_DEFAULT)] Event 84d42d93_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.768 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(18dc1232_DEFAULT)] Event 18dc1232_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.855 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ed8df7fe_DEFAULT)] Event ed8df7fe_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.970 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e6f0a262_DEFAULT)] Event e6f0a262_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:57:52.504 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 6be4c23b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:57:52.630 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1d24356d_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:59:06.351 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409146351 start=1632408938667
2021/09/23 14:59:07.139 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409147139 start=1632408941309
2021/09/23 14:59:18.979 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(075e3ef5_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f107f7e0-af8d-4467-9040-897ee0e761dc=-101}
2021/09/23 14:59:19.277 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(0c13d584_DEFAULT)] Event 0c13d584_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:00:04.661 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(420b6066_DEFAULT)] Event 420b6066_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:04.834 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2fbf762e_DEFAULT)] Event 2fbf762e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:04.922 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(78ede6b2_DEFAULT)] Event 78ede6b2_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.035 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1e480059_DEFAULT)] Event 1e480059_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.210 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1cd26804_DEFAULT)] Event 1cd26804_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.355 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f1bb47db_DEFAULT)] Event f1bb47db_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.455 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(590b10d6_DEFAULT)] Event 590b10d6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.579 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c94697ab_DEFAULT)] Event c94697ab_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.681 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(33eeaf99_DEFAULT)] Event 33eeaf99_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.815 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2bbfca27_DEFAULT)] Event 2bbfca27_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.961 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e4869a21_DEFAULT)] Event e4869a21_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:06.059 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(611e1452_DEFAULT)] Event 611e1452_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:02:17.827 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409337826 start=1632409149376
2021/09/23 15:02:21.397 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409341397 start=1632409150144
2021/09/23 15:03:30.974 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c1ca9535_DEFAULT)] Event c1ca9535_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.073 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0eba862f_DEFAULT)] Event 0eba862f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.249 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0e17fedc_DEFAULT)] Event 0e17fedc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.528 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(175ad4f3_DEFAULT)] Event 175ad4f3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.633 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2e581f07_DEFAULT)] Event 2e581f07_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.732 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d15fe12c_DEFAULT)] Event d15fe12c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:36.986 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4976150f_DEFAULT)] Event 4976150f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:37.099 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ce724eb0_DEFAULT)] Event ce724eb0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:04:21.947 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(ce79c31a_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d1a9bafa-373f-446e-9708-7c9684e539ca=-101}
2021/09/23 15:04:22.145 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(591553fd_DEFAULT)] Event 591553fd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:05:43.828 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409543828 start=1632409340846
2021/09/23 15:05:44.527 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409544527 start=1632409346011
2021/09/23 15:06:24.684 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2769a31c_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/df71bbe2-2ac4-4b2c-9b3b-31888d4e3912=-101}
2021/09/23 15:06:24.685 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2769a31c_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/df71bbe2-2ac4-4b2c-9b3b-31888d4e3912=-101}
2021/09/23 15:06:24.853 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ec4c3ee0_DEFAULT)] Event ec4c3ee0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:11:01.240 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(434ee506_DEFAULT)] Event 434ee506_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:13:14.363 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(cf5f3456_DEFAULT)] Event cf5f3456_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:18:06.991 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event e30e8c91_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 15:20:17.211 WARN [SegmentCompletionFSM_spanEventView__3__33065__20210923T1511Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84659363493 now=1632410417211 start=1632410214302
2021/09/23 15:30:06.111 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(834e3694_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/57fa43d4-fd0b-48c1-8405-be3df7bd859d=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/78ba0c39-0aef-48c2-8b02-71635bef606a=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/faa647fc-faf2-4f67-a47c-f0fd1d4be264=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/23065e4a-07dd-4b15-b8f8-0ddcee662f97=-101}
2021/09/23 15:30:06.380 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(63e942a2_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/b1d22374-5f5b-4ac9-b651-b41792054808=-101, /pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/fba70cad-12a3-49a0-ac54-8cb7e49308aa=-101}
2021/09/23 15:30:10.726 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment rawTraceView__1__1458__20210923T1232Z of table rawTraceView_REALTIME has no replicas
2021/09/23 15:30:10.727 WARN [SegmentStatusChecker] [pool-9-thread-4] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 15:30:10.727 WARN [SegmentStatusChecker] [pool-9-thread-4] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:30:10.736 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__0__9858__20210923T1406Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.736 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__0__9859__20210923T1444Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__3__9910__20210923T1353Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Table serviceCallView_REALTIME has 3 segments with no online replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:30:10.877 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33509__20210923T1355Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33510__20210923T1441Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33511__20210923T1444Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33512__20210923T1447Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.883 WARN [SegmentStatusChecker] [pool-9-thread-4] Table spanEventView_REALTIME has 30 segments with no online replicas
2021/09/23 15:30:10.883 WARN [SegmentStatusChecker] [pool-9-thread-4] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 15:32:04.930 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust edited a comment on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust edited a comment on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926046419
^The above chart is the incoming rate. There were no OOMs in the pods. Ingestion went back to normal once they were restarted. There were no conspicuous events that would lead to this sudden drop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin edited a comment on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin edited a comment on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926717352
Thanks for the information. This is quite a complex issue. What that stack trace means is that some call to `Unsafe.get` or `Unsafe.put` led to a SIGBUS error, which was caught asynchronously, meaning the frame in the error may or may not be directly related to the underlying cause. The underlying cause will be related to memory mapping, probably an OOM writing to `tmpfs`. There's a good explanation to refer to [here](https://stackoverflow.com/questions/45536049/java-lang-internalerror-a-fault-occurred-in-a-recent-unsafe-memory-access-opera).
We need to reproduce this, but if it happens again, you can figure out how much `tmpfs` is used by running `df -h` and looking for the row starting with `tmpfs`. If it happens again and `Use%` is high , then `tmpfs` OOM is likely to be the cause. If the risk of this happening again is too high, I recommend using more RAM until we have reproduced the issue and improved memory efficiency.
To help reproduce the issue, it would be good to get a redacted copy of your table config so we can see which indexes etc. you are using. This will narrow the search for the culprit. Also, knowing `-Xmx` and `-XX:MaxDirectMemorySize` values would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926093836
Hi @suddendust could you share information about your environment please? Pinot version, OS version, JVM version, instance type? Ping me in the pinot slack if you want to share this info privately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-927709836
@suddendust
> Hi @richardstartin thanks for the thorough analysis. So just to pin the speculated root-cause, a SIGBUS happened while consuming a real-time segment for some reason (probably high tmpfs usage) and therefore, the ingestion dropped to 0 and no new segments were generated. Is that correct?
Yes, the difficult part is the error is thrown asynchronously when the SIGBUS is handled, so pinpointing the precise root cause is hard. The root cause will certainly not be the in dictionary stack frame where the error is reported. Another way to cause this would be to write to a `MappedByteBuffer` after truncating the `RandomAccessFile` but I have audited the codebase and do not think it is possible for this to happen. If you have any of your own code which could do this (seems unlikely) it would be another possible culprit.
Thanks for the table config and memory settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-927360401
Hi @richardstartin thanks for the thorough analysis. So just to pin the speculated root-cause, a SIGBUS happened while consuming a real-time segment for some reason (probably high `tmpfs` usage) and therefore, the ingestion dropped to 0 and no new segments were generated. Is that correct?
Let me get back to you with the required info first thing tomorrow morning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926717352
Thanks for the information. This is quite a complex issue. What that stack trace means is that some call to `Unsafe.get` or `Unsafe.put` led to a SIGBUS error, which was caught asynchronously, meaning the frame in the error may or may be directly related to the underlying cause. The underlying cause will be related to memory mapping, probably an OOM writing to `tmpfs`. There's a good explanation to refer to [here](https://stackoverflow.com/questions/45536049/java-lang-internalerror-a-fault-occurred-in-a-recent-unsafe-memory-access-opera).
We need to reproduce this, but if it happens again, you can figure out how much `tmpfs` is used by running `df -h` and looking for the row starting with `tmpfs`. If it happens again and `Use%` is high , then `tmpfs` OOM is likely to be the cause. If the risk of this happening again is too high, I recommend using more RAM until we have reproduced the issue and improved memory efficiency.
To help reproduce the issue, it would be good to get a redacted copy of your table config so we can see which indexes etc. you are using. This will narrow the search for the culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926045579
Some metrics:
<img width="1358" alt="Screenshot 2021-09-23 at 9 19 12 PM" src="https://user-images.githubusercontent.com/84911643/134561498-62e7a31c-354d-4471-ac6f-6fff8f66161a.png">
<img width="1380" alt="Screenshot 2021-09-23 at 9 20 51 PM" src="https://user-images.githubusercontent.com/84911643/134561517-b50fa264-06db-4888-ac19-14a978a0053c.png">
<img width="688" alt="Screenshot 2021-09-23 at 9 21 37 PM" src="https://user-images.githubusercontent.com/84911643/134561521-443a05c2-91f2-4880-84c6-179a2c46a3b8.png">
<img width="680" alt="Screenshot 2021-09-23 at 9 21 52 PM" src="https://user-images.githubusercontent.com/84911643/134561526-ae29f825-f020-4b89-9818-9c623bef9b2f.png">
<img width="686" alt="Screenshot 2021-09-23 at 9 22 12 PM" src="https://user-images.githubusercontent.com/84911643/134561532-e9ecbd00-7bd1-4418-86f5-d47fb47e2393.png">
<img width="686" alt="Screenshot 2021-09-23 at 9 49 21 PM" src="https://user-images.githubusercontent.com/84911643/134561536-f8303497-8b8d-4abe-85f5-c03214c6f1ac.png">
<img width="1363" alt="Screenshot 2021-09-23 at 9 52 45 PM" src="https://user-images.githubusercontent.com/84911643/134561542-730ea037-5b0e-4176-be7a-f34f7806636f.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926405315
@richardstartin here's the env details:
1. Pinot Version: 0.7.1.
2. JVM Version: 1.8.
3. Instance Type: m5.4xlarge
4. OS Image: Ubuntu 18.04.5 LTS, Operating System: linux, Architecture: amd64, Container Runtime Version: docker://19.3.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-927600086
Hi @richardstartin here's the request information:
1. Table Configuration [redacted]
```
{
"REALTIME": {
"tableName": "",
"tableType": "REALTIME",
"segmentsConfig": {
"timeType": "MILLISECONDS",
"schemaName": "",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "7",
"timeColumnName": "start_time_millis",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"replicasPerPartition": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "",
"tagOverrideConfig": {
"realtimeConsuming": "",
"realtimeCompleted": ""
}
},
"tableIndexConfig": {
"loadMode": "MMAP",
"invertedIndexColumns": [
"someCol"
],
"rangeIndexColumns": [
"someCol"
],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "LowLevel",
"stream.kafka.topic.name": "",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "",
"security.protocol": "",
"ssl.truststore.location": "",
"ssl.keystore.location": "",
"ssl.truststore.password": "",
"ssl.keystore.password": "",
"ssl.endpoint.identification.algorithm": "",
"stream.kafka.decoder.prop.schema.registry.url": "",
"stream.kafka.decoder.prop.schema.registry.rest.url": "",
"realtime.segment.flush.threshold.rows": "6000000",
"realtime.segment.flush.threshold.size": "6000000",
"realtime.segment.flush.threshold.time": "1d",
"realtime.segment.flush.threshold.segment.size": "1G",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
},
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false
},
"metadata": {},
"isDimTable": false
}
}
```
2. -`Xmx`: 4G
3. We haven't configured `XX:MaxDirectMemorySize` explicitly, so I guess the default would be same as 4G?
All other tables have similar config (we have seven tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926062770
Thanks for sharing the issue. @richardstartin could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-937606802
@subintp can you share the log line which should be above this please?
```java
LOGGER.info("Allocating {} bytes for: {}", _chunkSizeInBytes, _allocationContext);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-975797635
@suddendust can you close this if the problem has been solved please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-979703727
@richardstartin thanks for the fixes and all the quick help :) Closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] subintp commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
subintp commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-937589315
```
2021/10/07 08:49:31.176 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Exception while executing a state transition task rawTraceView__3__1573__20211007T0849Z
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.logging.log4j.core.AbstractLifeCycle.isStarted(AbstractLifeCycle.java:86) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.ensureAppenderStarted(AppenderControl.java:134) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:127) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2025) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1898) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:189) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:206) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:300) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1270) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:314) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:138) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:88) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
... 12 more
2021/10/07 08:49:31.186 ERROR [StateModel] [HelixTaskExecutor-message_handle_thread] Default rollback method invoked on error. Error Code: ERROR
2021/10/07 08:49:31.191 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread] Message execution failed. msgId: b625abf7-c975-4a77-84d8-e905eb461ad1, errorMsg: java.lang.reflect.InvocationTargetException
2021/10/07 08:49:31.199 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Skip internal error. errCode: ERROR, errMsg: null
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926046419
^The above chart is the incoming rate. There were no OOMs in the pods. Ingestion went back to normal once they were restarted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust closed issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust closed issue #7476:
URL: https://github.com/apache/pinot/issues/7476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust edited a comment on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust edited a comment on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926405315
@richardstartin here're the env details:
1. Pinot Version: 0.7.1.
2. JVM Version: 1.8.
3. Instance Type: m5.4xlarge
4. OS Image: Ubuntu 18.04.5 LTS, Operating System: linux, Architecture: amd64, Container Runtime Version: docker://19.3.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust edited a comment on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust edited a comment on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-927600086
Hi @richardstartin here's the request information:
1. Table Configuration [redacted]
```
{
"REALTIME": {
"tableName": "",
"tableType": "REALTIME",
"segmentsConfig": {
"timeType": "MILLISECONDS",
"schemaName": "",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "7",
"timeColumnName": "start_time_millis",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"replicasPerPartition": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "",
"tagOverrideConfig": {
"realtimeConsuming": "",
"realtimeCompleted": ""
}
},
"tableIndexConfig": {
"loadMode": "MMAP",
"invertedIndexColumns": [
"someCol"
],
"rangeIndexColumns": [
"someCol"
],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "LowLevel",
"stream.kafka.topic.name": "",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "",
"security.protocol": "",
"ssl.truststore.location": "",
"ssl.keystore.location": "",
"ssl.truststore.password": "",
"ssl.keystore.password": "",
"ssl.endpoint.identification.algorithm": "",
"stream.kafka.decoder.prop.schema.registry.url": "",
"stream.kafka.decoder.prop.schema.registry.rest.url": "",
"realtime.segment.flush.threshold.rows": "6000000",
"realtime.segment.flush.threshold.size": "6000000",
"realtime.segment.flush.threshold.time": "1d",
"realtime.segment.flush.threshold.segment.size": "1G",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
},
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false
},
"metadata": {},
"isDimTable": false
}
}
```
All other tables have similar config (we have seven tables).
2. `Xmx`: 4G
3. `XX:MaxDirectMemorySize`: This hasn't been configured, so it would default to [85%](https://www.eclipse.org/openj9/docs/xxmaxdirectmemorysize/) of 4G?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-927937150
@richardstartin apologies, I gave you the `xmx` for the broker earlier. The correct `xmx` value is `10G`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] subintp commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
subintp commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-937589315
```
2021/10/07 08:49:31.176 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Exception while executing a state transition task rawTraceView__3__1573__20211007T0849Z
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.logging.log4j.core.AbstractLifeCycle.isStarted(AbstractLifeCycle.java:86) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.ensureAppenderStarted(AppenderControl.java:134) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:127) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2025) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1898) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:189) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:206) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:300) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1270) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:314) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:138) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:88) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
... 12 more
2021/10/07 08:49:31.186 ERROR [StateModel] [HelixTaskExecutor-message_handle_thread] Default rollback method invoked on error. Error Code: ERROR
2021/10/07 08:49:31.191 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread] Message execution failed. msgId: b625abf7-c975-4a77-84d8-e905eb461ad1, errorMsg: java.lang.reflect.InvocationTargetException
2021/10/07 08:49:31.199 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Skip internal error. errCode: ERROR, errMsg: null
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-937606802
@subintp can you share the log line which should be above this please?
```java
LOGGER.info("Allocating {} bytes for: {}", _chunkSizeInBytes, _allocationContext);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin edited a comment on issue #7476: Pinot Ingestion Rate Sudden Drop to 0
Posted by GitBox <gi...@apache.org>.
richardstartin edited a comment on issue #7476:
URL: https://github.com/apache/pinot/issues/7476#issuecomment-926717352
Thanks for the information. This is quite a complex issue. What that stack trace means is that some call to `Unsafe.get` or `Unsafe.put` led to a SIGBUS error, which was caught asynchronously, meaning the frame in the error may or may not be directly related to the underlying cause. The underlying cause will be related to memory mapping, possibly an OOM writing to `tmpfs`. There's a good explanation to refer to [here](https://stackoverflow.com/questions/45536049/java-lang-internalerror-a-fault-occurred-in-a-recent-unsafe-memory-access-opera).
We need to reproduce this, but if it happens again, you can figure out how much `tmpfs` is used by running `df -h` and looking for the row starting with `tmpfs`. If it happens again and `Use%` is high , then `tmpfs` OOM is likely to be the cause. If the risk of this happening again is too high, I recommend using more RAM until we have reproduced the issue and improved memory efficiency.
To help reproduce the issue, it would be good to get a redacted copy of your table config so we can see which indexes etc. you are using. This will narrow the search for the culprit. Also, knowing `-Xmx` and `-XX:MaxDirectMemorySize` values would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org