You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/16 19:17:49 UTC
[GitHub] [incubator-pinot] pabrahamusa opened a new issue #6449: Errors after Pinot server ran out of max map count
pabrahamusa opened a new issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449
My EKS node running pinot server ran out of map areas and I bumped it to 262144 from default.
When the error came pinot stopped consuming segments. I updated the max map in os and restarted the server. After starting
the realtime table was showing Bad status. However it recovered eventually. I am seeing following message after that and
wondering is this due to some hidden problem somewhere in pinot server ?
```
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
```
```
my-pinot-server-1 server 2021/01/16 18:50:51.938 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__162__20210112T1932Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__163__20210112T1935Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__164__20210112T1937Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__165__20210112T1940Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__166__20210112T1944Z
my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__167__20210112T1947Z
my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__168__20210112T1950Z
my-pinot-server-1 server 2021/01/16 18:50:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__169__20210112T1953Z
n config.
```
```
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.autotune.initialRows' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'streamType' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.zk.broker.url' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
```
```
my-pinot-server-1 server 2021-01-16T19:01:25.952+0000: 672.343: Total time for which application threads were stopped: 0.0003745 seconds, Stopping threads took: 0.0000540 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.158+0000: 672.549: Application time: 0.2053891 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.159+0000: 672.550: Total time for which application threads were stopped: 0.0015166 seconds, Stopping threads took: 0.0000398 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.227+0000: 672.618: Application time: 0.0673477 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.228+0000: 672.619: Total time for which application threads were stopped: 0.0015950 seconds, Stopping threads took: 0.0000456 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.242+0000: 672.633: Application time: 0.0142821 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.243+0000: 672.634: Total time for which application threads were stopped: 0.0003874 seconds, Stopping threads took: 0.0000330 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Application time: 0.0067833 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Total time for which application threads were stopped: 0.0003493 seconds, Stopping threads took: 0.0000236 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.251+0000: 672.642: Application time: 0.0010896 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.253+0000: 672.643: Total time for which application threads were stopped: 0.0014404 seconds, Stopping threads took: 0.0000105 seconds
my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Application time: 1.0000987 seconds
my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Total time for which application threads were stopped: 0.0004062 seconds, Stopping threads took: 0.0000532 seconds
my-pinot-server-1 server 2021-01-16T19:01:30.253+0000: 676.644: Application time: 3.0002441 seconds
my-pinot-server-1 server 2021-01-16T19:01:30.254+0000: 676.645: Total time for which application threads were stopped: 0.0003946 seconds, Stopping threads took: 0.0000391 seconds
my-pinot-server-1 server 2021-01-16T19:02:17.596+0000: 723.987: Application time: 47.3426464 seconds
my-pinot-server-1 server 2021-01-16T19:02:17.597+0000: 723.988: Total time for which application threads were stopped: 0.0004295 seconds, Stopping threads took: 0.0000559 seconds
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765046731
Can you check if the controller deep store is full? It may prevent the consuming segments to be uploaded to it, which stops the consumption.
Also, from the log, seems that you have a lot of segments, each with a small size like 3.6MB, what's the segment flush size you configured? I feel we can at least make the flush size 50 times to reduce the number of segments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764822003
These are my Cluster info
Number of Segments : 23114 / 23115
Status : BAD (in the UI ) This is a bug?
Estimated Size : 84312409908
Reported Size: 84312409908
To Start the ingestion back I tried to restart controller, broker , pinot server one by one to avoid outage. However that didn't work. So i have to start all controllers, brokers and servers at once. After that ingestion resumed and caught up with kafka.
Do we have to do any manual or scripted restarts when system memory usage goes high for a long time?
Or do autoscaling works, I only have single partition at the moment so I assume autoscaling will not help here?
Not idea what is going on with the cluster..!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] subourbonite commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
subourbonite commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765114870
You can't have two controllers with separate storage. You can run multiple controllers, but they must have a consistent view of the data (S3, EFS, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764481860
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-767304859
you can follow this doc to add configs for s3.
https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764822003
These are my Cluster info
Number of Segments : 23114 / 23115
Status : BAD (in the UI ) This is a bug?
Estimated Size : 84312409908
Reported Size: 84312409908
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-768184800
> @fx19880617 , I followed the steps and tried to modify JAVA_OPTS to add the se plugin. But it does break kafka. I am seeing the following error.
>
> [main] {"code":500,"error":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory"}
Can you check if you set the plugin directory in the JAVA_OPTS? `-Dplugins.dir=/opt/pinot/plugins`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] subourbonite commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
subourbonite commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765114870
You can't have two controllers with separate storage. You can run multiple controllers, but they must have a consistent view of the data (S3, EFS, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-769332486
Thanks @fx19880617 , will try the CRON minion and see. Also as @subourbonite mentioned I mapped the stateful set for controller to use single pvc mount. So controller data is shared across all controller pods. Also I mapped the controller to use an EFS with no restriction in data size. Now my question is
As the controller pvc have a copy of all segments will it suffice as a backup if one of the Pinot Server SSD fails? If one of the Pinot Server SSD fails and if I replace the disk will the server catches up the lost segments from controller EFS automatically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764822003
These are my Cluster info
Number of Segments : 23114 / 23115
Status : BAD (in the UI ) This is a bug?
Estimated Size : 84312409908
Reported Size: 84312409908
To Start the ingestion back I tried to restart controller, broker , pinot server one by one to avoid outage. However that didn't work. So i have to start all controllers, brokers and servers at once. After that ingestion resumed and caught up with kafka.
Not idea what is going on with the cluster..!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-767304859
you can follow this doc to add configs for s3.
[Use S3 as deep store for Pinot](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot)
More ref:
[Pinot Amazon S3 filesystem](https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3)
[Ingest Parquet Files from S3 Using Spark](https://docs.pinot.apache.org/users/tutorials/ingest-parquet-files-from-s3-using-spark)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-767933847
@fx19880617 , Other than adding configs to controller and server I can see a Job spec as well. Is Job spec mandatory? I was expecting controller to push segment directly to s3 and download directly from s3 whenever required without any job. Is it possible to implement S3 backend without Job?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
@fx19880617 Thanks for the details. I just keep monitoring the system and the memory for the System was going through the edge. I noticed that once it happens and If i leave it for say couple of days following errors will appear and Pinot server will stop ingestion completely. Is there any way to fix this issue?
```
sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.5, Avg: 1.6, Max: 1.8, Diff: 0.3, Sum: 9.4]
2021/01/21 17:02:38.123 ERROR [ServerSegmentCompletionProtocolHandler] [sblog__0__32545__20210121T1702Z] Could not send request http://sb-pinot-controller-0.sb-pinot-controller-headless.logging.svc.cluster.local:9000/segmentCommit?name=sblog__0__32545__20210121T1702Z&offset=44504082&instance=Server_sb-pinot-server-1.sb-pinot-server-headless.logging.svc.cluster.local_8098&buildTimeMillis=191&memoryUsedBytes=256504&segmentSizeBytes=2763568&rowCount=1000&streamPartitionMsgOffset=44504082
sb-pinot-server-1 sb-pinot-server-0server server [Update RS (ms): Min: 5.6, Avg: 5.7, Max: 5.7, Diff: 0.1, Sum: 34.1]
sb-pinot-controller-1 shaded.org.apache.http.NoHttpResponseException: sb-pinot-controller-0.sb-pinot-controller-headless.logging.svc.cluster.local:9000 failed to respond
sb-pinot-server-1sb-pinot-server-0 sb-pinot-broker-0 server [Processed Buffers: Min: 17, Avg: 27.3, Max: 34, Diff: 17, Sum: 164]
at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-a36d1c04c720c0f47dfec078c24c84c98ed665c4]
sb-pinot-broker-0 sb-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
broker [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-1 sb-pinot-broker-0 server broker [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 2]
sb-pinot-broker-0 sb-pinot-controller-1 server [Choose CSet: 0.0 ms]
sb-pinot-server-0 broker controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
sb-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
... 16 more
sb-pinot-controller-1server [Ref Proc: 3.4 ms]
controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-server-1 server 2021/01/21 17:04:50.874 ERROR [LLRealtimeSegmentDataManager_sblog__0__32545__20210121T1702Z] [sblog__0__32545__20210121T1702Z] Holding after response from Controller: {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"NOT_SENT","offset":-1}
sb-pinot-server-1 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
server sb-pinot-server-0 server [Ref Enq: 0.0 ms]
sb-pinot-server-1 controller 2021-01-21T17:25:03.929+0000: 1350.323: Application time: 0.0590331 seconds
sb-pinot-server-0 sb-pinot-controller-1 server controller 2021-01-21T17:25:03.930+0000: 1350.323: [GC pause (G1 Evacuation Pause) (mixed), 0.0406644 secs]
server [Times: user=0.26 sys=0.00, real=0.05 secs]
sb-pinot-server-0 2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29726__20210121T1102Z
server 2021-01-21T17:25:05.778+0000: 166.983: Total time for which application threads were stopped: 0.0500843 seconds, Stopping threads took: 0.0000282 seconds
sb-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: [GC concurrent-mark-end, 1.4470120 secs]
sb-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: Application time: 0.4400702 seconds
sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29727__20210121T1102Z
sb-pinot-server-1 server 2021-01-21T17:25:06.219+0000: 167.424: [GC remark 2021-01-21T17:25:06.219+0000: 167.424: [Finalize Marking, 0.0002085 secs] 2021-01-21T17:25:06.219+0000: 167.424: [GC ref-proc, 0.0851322 secs] 2021-01-21T17:25:06.304+0000: 167.509: [Unloading, 0.0080895 secs], 0.0963586 secs]
2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29728__20210121T1102Z
sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29729__20210121T1102Z
sb-pinot-server-1sb-pinot-server-0 server sb-pinot-controller-1 [Times: user=0.11 sys=0.04, real=0.10 secs]
sb-pinot-server-0server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29730__20210121T1102Z
sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29731__20210121T1102Z
sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29732__20210121T1102Z
controller [Parallel Time: 39.8 ms, GC Workers: 4]
sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350323.3, Avg: 1350323.3, Max: 1350323.3, Diff: 0.0]
server 2021-01-21T17:25:06.315+0000: 167.520: Total time for which application threads were stopped: 0.0972538 seconds, Stopping threads took: 0.0003432 seconds
sb-pinot-server-0 server sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 0.9, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.0]
sb-pinot-controller-1 sb-pinot-server-1 2021-01-21T17:25:06.315+0000: 167.520: Application time: 0.0001148 seconds
servercontroller [Update RS (ms): Min: 15.6, Avg: 15.6, Max: 15.6, Diff: 0.0, Sum: 62.4]
sb-pinot-controller-1 sb-pinot-server-0 servercontroller [Processed Buffers: Min: 76, Avg: 77.8, Max: 79, Diff: 3, Sum: 311]
sb-pinot-controller-1 controller [Scan RS (ms): Min: 1.7, Avg: 1.7, Max: 1.7, Diff: 0.1, Sum: 6.8]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 2021-01-21T17:25:06.316+0000: 167.521: [GC cleanup 3168M->3047M(4096M), 0.0040580 secs]
sb-pinot-server-0 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29733__20210121T1102Z
sb-pinot-server-1server [Times: user=0.02 sys=0.00, real=0.00 secs]
sb-pinot-server-0 server server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29734__20210121T1103Z
controller [Object Copy (ms): Min: 21.3, Avg: 21.4, Max: 21.5, Diff: 0.2, Sum: 85.6]
sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29735__20210121T1103Z
sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29736__20210121T1103Z
controller2021-01-21T17:25:06.320+0000: 167.525: Total time for which application threads were stopped: 0.0046889 seconds, Stopping threads took: 0.0001052 seconds
sb-pinot-server-0 sb-pinot-server-1server 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-start]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-controller-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29737__20210121T1103Z
controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29738__20210121T1103Z
sb-pinot-controller-1 server sb-pinot-server-1 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-end, 0.0001522 secs]
sb-pinot-server-0server server controller [GC Worker Total (ms): Min: 39.7, Avg: 39.7, Max: 39.7, Diff: 0.0, Sum: 158.9]
2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29739__20210121T1103Z
sb-pinot-controller-1 2021-01-21T17:25:06.929+0000: 168.134: Application time: 0.6090475 seconds
sb-pinot-server-0 sb-pinot-server-1 server controller [GC Worker End (ms): Min: 1350363.0, Avg: 1350363.0, Max: 1350363.0, Diff: 0.0]
sb-pinot-controller-1 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29740__20210121T1103Z
sb-pinot-server-1 serverserver 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29741__20210121T1103Z
sb-pinot-server-1 controller [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controller 2021-01-21T17:25:06.930+0000: 168.135: [GC pause (G1 Evacuation Pause) (young), 0.0357839 secs]
sb-pinot-server-0 server [Parallel Time: 32.1 ms, GC Workers: 6]
sb-pinot-server-0 server [GC Worker Start (ms): Min: 168134.9, Avg: 168134.9, Max: 168134.9, Diff: 0.1]
[Code Root Purge: 0.0 ms]
sb-pinot-controller-1sb-pinot-server-0 server controller [Clear CT: 0.1 ms]
sb-pinot-controller-1 controller [Other: 0.8 ms]
sb-pinot-controller-1 controller server 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29742__20210121T1104Z
sb-pinot-server-1 [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.5, Diff: 0.4, Sum: 7.9]
sb-pinot-server-0 server [Choose CSet: 0.1 ms]
sb-pinot-controller-1 controller [Ref Proc: 0.2 ms]
sb-pinot-controller-1 controller [Ref Enq: 0.0 ms]
sb-pinot-controller-1 controller server [Update RS (ms): Min: 14.8, Avg: 14.9, Max: 15.1, Diff: 0.3, Sum: 89.2]
sb-pinot-server-0 server [Processed Buffers: Min: 107, Avg: 122.2, Max: 132, Diff: 25, Sum: 733]
sb-pinot-server-0 server [Scan RS (ms): Min: 1.9, Avg: 2.1, Max: 2.2, Diff: 0.3, Sum: 12.5]
sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Redirty Cards: 0.2 ms]
sb-pinot-controller-1 controller 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29743__20210121T1104Z
sb-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29744__20210121T1104Z
[Humongous Register: 0.0 ms]
sb-pinot-controller-1 sb-pinot-server-0 server [Object Copy (ms): Min: 13.5, Avg: 13.7, Max: 13.8, Diff: 0.3, Sum: 82.1]
sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
controller [Humongous Reclaim: 0.0 ms]
sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29745__20210121T1104Z
sb-pinot-server-0 sb-pinot-server-1 [Free CSet: 0.1 ms]
sb-pinot-controller-1 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.3]
serversb-pinot-server-0controller server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29746__20210121T1104Z
sb-pinot-server-1 [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 466.6M(1024.0M)->412.2M(1024.0M)]
sb-pinot-controller-1 server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29747__20210121T1104Z
sb-pinot-server-1 server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29748__20210121T1105Z
[GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.1, Diff: 0.1, Sum: 192.1]
sb-pinot-server-1 sb-pinot-server-0 server 2021/01/21 17:22:51.947 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29749__20210121T1105Z
sb-pinot-server-1 server server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29750__20210121T1105Z
controller [Times: user=0.16 sys=0.00, real=0.04 secs]
[GC Worker End (ms): Min: 168166.9, Avg: 168166.9, Max: 168166.9, Diff: 0.1]
sb-pinot-controller-1 sb-pinot-server-1 controller 2021-01-21T17:25:03.971+0000: 1350.364: Total time for which application threads were stopped: 0.0412316 seconds, Stopping threads took: 0.0000680 seconds
sb-pinot-controller-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29751__20210121T1105Z
sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29752__20210121T1105Z
sb-pinot-server-1 serversb-pinot-server-0 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29753__20210121T1105Z
server [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 controller 2021-01-21T17:25:04.025+0000: 1350.418: Application time: 0.0544185 seconds
sb-pinot-controller-1 controllerserver 2021-01-21T17:25:04.025+0000: 1350.419: [GC pause (G1 Evacuation Pause) (mixed), 0.0437685 secs]
[Code Root Purge: 0.0 ms]
sb-pinot-server-0 server [Clear CT: 0.5 ms]
sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29754__20210121T1105Z
sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29755__20210121T1105Z
sb-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29756__20210121T1106Z
sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29757__20210121T1106Z
server [Other: 3.2 ms]
sb-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29758__20210121T1106Z
sb-pinot-server-1 sb-pinot-server-0 controller server [Choose CSet: 0.0 ms]
[Parallel Time: 43.0 ms, GC Workers: 4]
sb-pinot-server-0 server [Ref Proc: 1.6 ms]
serversb-pinot-server-0 server [Ref Enq: 0.0 ms]
2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29759__20210121T1106Z
sb-pinot-server-1 sb-pinot-server-0 serverserver [Redirty Cards: 0.1 ms]
sb-pinot-controller-1 sb-pinot-server-0 controller 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29760__20210121T1106Z
sb-pinot-server-1 server 2021/01/21 17:22:51.950 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29761__20210121T1106Z
sb-pinot-server-1 [GC Worker Start (ms): Min: 1350418.9, Avg: 1350418.9, Max: 1350418.9, Diff: 0.0]
sb-pinot-controller-1 server controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.1]
sb-pinot-controller-1 controller [Update RS (ms): Min: 23.6, Avg: 23.6, Max: 23.6, Diff: 0.0, Sum: 94.5]
sb-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 106.8, Max: 109, Diff: 6, Sum: 427]
server2021/01/21 17:22:51.950 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29762__20210121T1106Z
sb-pinot-server-1 server 2021/01/21 17:22:51.951 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29763__20210121T1107Z
sb-pinot-server-1 [Humongous Register: 0.2 ms]
server 2021/01/21 17:22:51.951 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29764__20210121T1107Z
sb-pinot-server-1 sb-pinot-controller-1 server controller [Scan RS (ms): Min: 0.2, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.8]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29765__20210121T1107Z
controller sb-pinot-server-0 sb-pinot-server-1 server server [Humongous Reclaim: 0.3 ms]
[Object Copy (ms): Min: 18.0, Avg: 18.1, Max: 18.2, Diff: 0.1, Sum: 72.4]
sb-pinot-server-0 sb-pinot-controller-1 controller server [Free CSet: 0.8 ms]
sb-pinot-server-0 [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29766__20210121T1107Z
sb-pinot-server-1 server server2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29767__20210121T1107Z
sb-pinot-server-1 controller server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29768__20210121T1107Z
sb-pinot-server-1 [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 43.0, Avg: 43.0, Max: 43.0, Diff: 0.0, Sum: 171.9]
server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29769__20210121T1107Z
sb-pinot-server-1 [Eden: 900.0M(900.0M)->0.0B(167.0M) Survivors: 32768.0K->37888.0K Heap: 3705.3M(4096.0M)->2630.9M(4096.0M)]
sb-pinot-server-0 server sb-pinot-controller-1 controller server [GC Worker End (ms): Min: 1350461.9, Avg: 1350461.9, Max: 1350461.9, Diff: 0.0]
sb-pinot-controller-1 controller 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29770__20210121T1108Z
sb-pinot-server-1 [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
sb-pinot-controller-1 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29771__20210121T1108Z
sb-pinot-server-1 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29772__20210121T1108Z
[Times: user=0.19 sys=0.00, real=0.03 secs]
sb-pinot-server-1sb-pinot-server-0 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29773__20210121T1108Z
serversb-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29774__20210121T1108Z
sb-pinot-server-1 controllerserver 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29775__20210121T1108Z
[Clear CT: 0.1 ms]
sb-pinot-controller-1 controller 2021-01-21T17:25:06.966+0000: 168.170: Total time for which application threads were stopped: 0.0366911 seconds, Stopping threads took: 0.0002408 seconds
sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29776__20210121T1108Z
sb-pinot-server-1 server [Other: 0.6 ms]
sb-pinot-controller-1 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29777__20210121T1109Z
sb-pinot-server-1 controller [Choose CSet: 0.0 ms]
sb-pinot-controller-1 server controller 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29778__20210121T1109Z
sb-pinot-server-1 2021-01-21T17:25:07.002+0000: 168.207: Application time: 0.0364979 seconds
sb-pinot-server-0 server 2021-01-21T17:25:07.004+0000: 168.208: [GC pause (G1 Evacuation Pause) (mixed), 0.0506602 secs]
sb-pinot-server-0 server [Parallel Time: 47.3 ms, GC Workers: 6]
[Ref Proc: 0.1 ms]
sb-pinot-server-0 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29779__20210121T1109Z
server [GC Worker Start (ms): Min: 168208.9, Avg: 168208.9, Max: 168208.9, Diff: 0.1]
sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.4, Max: 1.6, Diff: 0.3, Sum: 8.3]
sb-pinot-server-0 sb-pinot-controller-1 controller [Ref Enq: 0.0 ms]
sb-pinot-controller-1 controller [Redirty Cards: 0.1 ms]
sb-pinot-controller-1 controllersb-pinot-server-1 [Humongous Register: 0.0 ms]
sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
sb-pinot-controller-1server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29780__20210121T1109Z
sb-pinot-server-1 server server [Update RS (ms): Min: 3.5, Avg: 12.9, Max: 31.6, Diff: 28.1, Sum: 77.6]
sb-pinot-server-0 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29781__20210121T1109Z
sb-pinot-server-1 controller server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29782__20210121T1109Z
sb-pinot-server-1 serverserver [Processed Buffers: Min: 6, Avg: 14.3, Max: 31, Diff: 25, Sum: 86]
2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29783__20210121T1109Z
sb-pinot-server-0 server [Scan RS (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: 0.8, Sum: 2.8]
sb-pinot-server-1sb-pinot-server-0 [Free CSet: 0.1 ms]
sb-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 470.8M(1024.0M)->444.5M(1024.0M)]
sb-pinot-controller-1 controller [Times: user=0.17 sys=0.00, real=0.05 secs]
sb-pinot-controller-1server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 controller server [Object Copy (ms): Min: 14.1, Avg: 32.3, Max: 41.4, Diff: 27.4, Sum: 193.9]
sb-pinot-server-0 2021-01-21T17:25:04.069+0000: 1350.463: Total time for which application threads were stopped: 0.0443644 seconds, Stopping threads took: 0.0001010 seconds
sb-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29784__20210121T1109Z
sb-pinot-server-1server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29785__20210121T1110Z
sb-pinot-server-1controller server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29786__20210121T1110Z
sb-pinot-server-1 serversb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29787__20210121T1110Z
sb-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29788__20210121T1110Z
sb-pinot-server-1 2021-01-21T17:25:04.093+0000: 1350.486: Application time: 0.0234818 seconds
sb-pinot-controller-1 [GC Worker Total (ms): Min: 47.1, Avg: 47.1, Max: 47.1, Diff: 0.1, Sum: 282.8]
sb-pinot-server-0 server [GC Worker End (ms): Min: 168256.0, Avg: 168256.0, Max: 168256.1, Diff: 0.0]
sb-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29789__20210121T1110Z
sb-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29790__20210121T1110Z
controller sb-pinot-server-1 server 2021-01-21T17:25:04.093+0000: 1350.487: Total time for which application threads were stopped: 0.0004505 seconds, Stopping threads took: 0.0000400 seconds
sb-pinot-controller-1 server [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 server [Code Root Purge: 0.0 ms]
sb-pinot-server-0 server [Clear CT: 0.5 ms]
sb-pinot-server-0 server [Other: 2.9 ms]
sb-pinot-server-0 server [Choose CSet: 0.2 ms]
sb-pinot-server-0 server [Ref Proc: 1.1 ms]
sb-pinot-server-02021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29791__20210121T1110Z
sb-pinot-server-1 controller 2021-01-21T17:25:04.093+0000: 1350.487: Application time: 0.0000469 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.094+0000: 1350.487: [GC pause (GCLocker Initiated GC) (young), 0.0285995 secs]
sb-pinot-controller-1 server controller [Parallel Time: 27.9 ms, GC Workers: 4]
sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350487.1, Avg: 1350487.1, Max: 1350487.1, Diff: 0.0]
2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29792__20210121T1111Z
sb-pinot-server-1 server [Ref Enq: 0.0 ms]
sb-pinot-server-0 sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.2, Sum: 4.1]
server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29793__20210121T1111Z
sb-pinot-server-1 server sb-pinot-controller-1 server controller [Update RS (ms): Min: 21.6, Avg: 21.7, Max: 21.9, Diff: 0.2, Sum: 86.8]
2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29794__20210121T1111Z
sb-pinot-controller-1 sb-pinot-server-1 controller [Processed Buffers: Min: 85, Avg: 88.0, Max: 90, Diff: 5, Sum: 352]
sb-pinot-controller-1 [Redirty Cards: 0.1 ms]
controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29795__20210121T1111Z
sb-pinot-controller-1 sb-pinot-server-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29796__20210121T1111Z
sb-pinot-server-1 server 2021/01/21 17:22:51.955 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29797__20210121T1111Z
sb-pinot-server-1 server sb-pinot-server-0 controllerserver [Humongous Register: 0.1 ms]
2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29798__20210121T1111Z
sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29799__20210121T1111Z
sb-pinot-server-1 sb-pinot-server-0 server [Object Copy (ms): Min: 4.9, Avg: 5.1, Max: 5.2, Diff: 0.3, Sum: 20.4]
[Humongous Reclaim: 0.1 ms]
sb-pinot-controller-1 sb-pinot-server-0 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29800__20210121T1111Z
sb-pinot-server-1 server server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29801__20210121T1111Z
sb-pinot-server-1 server sb-pinot-controller-1 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29802__20210121T1112Z
controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-server-1 [Free CSet: 0.9 ms]
sb-pinot-server-0 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29803__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29804__20210121T1112Z
sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29805__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29806__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29807__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29808__20210121T1112Z
sb-pinot-server-1 server server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29809__20210121T1112Z
controller [GC Worker Total (ms): Min: 27.8, Avg: 27.9, Max: 27.9, Diff: 0.0, Sum: 111.4]
sb-pinot-controller-1 [Eden: 167.0M(167.0M)->0.0B(200.0M) Survivors: 37888.0K->4096.0K Heap: 2797.9M(4096.0M)->2222.6M(4096.0M)]
controller [GC Worker End (ms): Min: 1350515.0, Avg: 1350515.0, Max: 1350515.0, Diff: 0.0]
sb-pinot-controller-1 sb-pinot-server-0 server [Times: user=0.14 sys=0.00, real=0.05 secs]
sb-pinot-server-0 controller [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
sb-pinot-controller-1 sb-pinot-server-1 controller [Clear CT: 0.1 ms]
sb-pinot-controller-1 controller [Other: 0.6 ms]
sb-pinot-controller-1 controller [Choose CSet: 0.0 ms]
server server 2021-01-21T17:25:07.054+0000: 168.259: Total time for which application threads were stopped: 0.0522104 seconds, Stopping threads took: 0.0008706 seconds
sb-pinot-server-0 server 2021-01-21T17:25:07.304+0000: 168.509: Application time: 0.2499433 seconds
sb-pinot-server-0 sb-pinot-controller-1 server 2021-01-21T17:25:07.305+0000: 168.510: [GC pause (G1 Evacuation Pause) (mixed), 0.0317508 secs]
controller [Ref Proc: 0.1 ms]
sb-pinot-controller-1 controller [Ref Enq: 0.0 ms]
sb-pinot-server-0 server sb-pinot-controller-1 controller [Redirty Cards: 0.1 ms]
[Parallel Time: 29.1 ms, GC Workers: 6]
sb-pinot-controller-1 controller 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29810__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29811__20210121T1112Z
[Humongous Register: 0.0 ms]
sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
sb-pinot-controller-1 controller sb-pinot-server-0 server [GC Worker Start (ms): Min: 168510.7, Avg: 168510.7, Max: 168510.7, Diff: 0.1]
sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.4, Diff: 0.2, Sum: 7.9]
sb-pinot-server-1 [Free CSet: 0.1 ms]
sb-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(91136.0K) Survivors: 7168.0K->2048.0K Heap: 493.8M(1024.0M)->450.0M(1024.0M)]
sb-pinot-server-0 sb-pinot-controller-1 server controller [Times: user=0.11 sys=0.00, real=0.03 secs]
sb-pinot-controller-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29812__20210121T1112Z
sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29813__20210121T1112Z
controller 2021-01-21T17:25:04.122+0000: 1350.516: Total time for which application threads were stopped: 0.0289920 seconds, Stopping threads took: 0.0000117 seconds
sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29814__20210121T1112Z
sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29815__20210121T1113Z
sb-pinot-server-1 [Update RS (ms): Min: 22.7, Avg: 22.7, Max: 22.8, Diff: 0.1, Sum: 136.4]
sb-pinot-server-0 server controller server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29816__20210121T1113Z
[Processed Buffers: Min: 90, Avg: 107.3, Max: 116, Diff: 26, Sum: 644]
sb-pinot-server-0 server [Scan RS (ms): Min: 1.2, Avg: 1.2, Max: 1.2, Diff: 0.1, Sum: 7.3]
sb-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29817__20210121T1113Z
sb-pinot-server-1 sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Object Copy (ms): Min: 3.5, Avg: 3.7, Max: 3.8, Diff: 0.2, Sum: 22.1]
server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29818__20210121T1113Z
sb-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29819__20210121T1113Z
sb-pinot-server-1 2021-01-21T17:25:04.125+0000: 1350.518: Application time: 0.0027123 seconds
sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller server 2021-01-21T17:25:04.125+0000: 1350.519: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0110860 secs]
sb-pinot-controller-1 sb-pinot-server-0 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29820__20210121T1113Z
server [Termination Attempts: Min: 1, Avg: 1.3, Max: 2, Diff: 1, Sum: 8]
sb-pinot-server-0 server sb-pinot-server-1 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
controller [Parallel Time: 10.5 ms, GC Workers: 4]
sb-pinot-controller-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29821__20210121T1114Z
sb-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29822__20210121T1114Z
controller [GC Worker Start (ms): Min: 1350519.1, Avg: 1350519.1, Max: 1350519.1, Diff: 0.0]
sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.2, Max: 1.4, Diff: 0.2, Sum: 5.0]
sb-pinot-controller-1 sb-pinot-server-1 sb-pinot-server-0 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29823__20210121T1114Z
sb-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29824__20210121T1114Z
serversb-pinot-server-1controller [Update RS (ms): Min: 8.3, Avg: 8.3, Max: 8.3, Diff: 0.0, Sum: 33.2]
sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 28.9, Avg: 29.0, Max: 29.0, Diff: 0.0, Sum: 173.8]
sb-pinot-server-0 [Processed Buffers: Min: 37, Avg: 38.8, Max: 41, Diff: 4, Sum: 155]
sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server [GC Worker End (ms): Min: 168539.7, Avg: 168539.7, Max: 168539.7, Diff: 0.0]
sb-pinot-server-0 sb-pinot-controller-1 controller [Object Copy (ms): Min: 0.7, Avg: 0.8, Max: 0.9, Diff: 0.2, Sum: 3.3]
sb-pinot-controller-1 servercontroller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 5]
sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29825__20210121T1114Z
server [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 server [Code Root Purge: 0.0 ms]
sb-pinot-server-0 [GC Worker Total (ms): Min: 10.4, Avg: 10.4, Max: 10.4, Diff: 0.0, Sum: 41.5]
sb-pinot-controller-1 sb-pinot-server-1 server controller [GC Worker End (ms): Min: 1350529.5, Avg: 1350529.5, Max: 1350529.5, Diff: 0.0]
server [Clear CT: 0.4 ms]
2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29826__20210121T1114Z
sb-pinot-server-0 sb-pinot-server-1 sb-pinot-controller-1 controller server [Code Root Fixup: 0.0 ms]
[Other: 2.3 ms]
sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
sb-pinot-controller-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29827__20210121T1114Z
sb-pinot-server-1 server sb-pinot-server-0 server [Choose CSet: 0.2 ms]
sb-pinot-server-0 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29828__20210121T1114Z
controller sb-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29829__20210121T1115Z
sb-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29830__20210121T1115Z
sb-pinot-server-1 server [Ref Proc: 0.3 ms]
[Clear CT: 0.1 ms]
sb-pinot-server-0 server sb-pinot-controller-1 controller [Other: 0.5 ms]
sb-pinot-controller-1 controller 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29831__20210121T1115Z
sb-pinot-server-1 serverserver [Ref Enq: 0.0 ms]
sb-pinot-server-0 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29832__20210121T1115Z
server sb-pinot-server-1 server 2021/01/21 17:22:51.961 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29833__20210121T1115Z
sb-pinot-server-1 server [Redirty Cards: 0.1 ms]
sb-pinot-server-0 [Choose CSet: 0.0 ms]
2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29834__20210121T1115Z
sb-pinot-server-1 server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29835__20210121T1115Z
sb-pinot-server-1 sb-pinot-controller-1 controller server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29836__20210121T1115Z
sb-pinot-server-1 server [Humongous Register: 0.1 ms]
server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29837__20210121T1115Z
sb-pinot-server-0 sb-pinot-server-1 serverserver 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29838__20210121T1116Z
sb-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29839__20210121T1116Z
sb-pinot-server-1 [Humongous Reclaim: 0.1 ms]
server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29840__20210121T1116Z
sb-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29841__20210121T1116Z
sb-pinot-server-0 [Ref Proc: 0.1 ms]
server [Free CSet: 1.1 ms]
sb-pinot-server-0 server [Eden: 200.0M(200.0M)->0.0B(195.0M) Survivors: 4096.0K->9216.0K Heap: 2471.9M(4096.0M)->1820.5M(4096.0M)]
sb-pinot-server-0 server sb-pinot-server-1 [Times: user=0.18 sys=0.01, real=0.03 secs]
server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29842__20210121T1116Z
sb-pinot-server-1 server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29843__20210121T1116Z
sb-pinot-controller-1 sb-pinot-server-0 sb-pinot-server-1server 2021-01-21T17:25:07.337+0000: 168.542: Total time for which application threads were stopped: 0.0329319 seconds, Stopping threads took: 0.0005259 seconds
sb-pinot-server-0 server controller [Ref Enq: 0.0 ms]
sb-pinot-controller-1server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29844__20210121T1116Z
sb-pinot-server-1 server controller 2021-01-21T17:25:07.632+0000: 168.836: Application time: 0.2943829 seconds
sb-pinot-server-0 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29845__20210121T1117Z
sb-pinot-server-1 server server 2021-01-21T17:25:07.632+0000: 168.837: [GC pause (G1 Evacuation Pause) (mixed), 0.0321493 secs]
[Redirty Cards: 0.1 ms]
sb-pinot-server-0 server [Parallel Time: 28.6 ms, GC Workers: 6]
sb-pinot-server-0 server [GC Worker Start (ms): Min: 168838.0, Avg: 168838.0, Max: 168838.0, Diff: 0.1]
sb-pinot-controller-1 controller [Humongous Register: 0.0 ms]
sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
sb-pinot-server-0 sb-pinot-controller-1 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.3, Max: 1.5, Diff: 0.2, Sum: 8.1]
sb-pinot-server-0 server [Update RS (ms): Min: 9.7, Avg: 9.8, Max: 9.8, Diff: 0.1, Sum: 58.6]
sb-pinot-server-0 controllerserver [Free CSet: 0.0 ms]
[Processed Buffers: Min: 67, Avg: 79.5, Max: 89, Diff: 22, Sum: 477]
2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29846__20210121T1117Z
sb-pinot-server-1 sb-pinot-server-0 sb-pinot-controller-1 server [Scan RS (ms): Min: 1.8, Avg: 1.8, Max: 1.9, Diff: 0.1, Sum: 10.9]
server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29847__20210121T1117Z
controller [Eden: 5120.0K(91136.0K)->0.0B(152.0M) Survivors: 2048.0K->3072.0K Heap: 461.0M(1024.0M)->453.5M(1024.0M)]
sb-pinot-server-0 sb-pinot-controller-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Object Copy (ms): Min: 15.4, Avg: 15.5, Max: 15.6, Diff: 0.2, Sum: 92.9]
sb-pinot-server-0 serversb-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29848__20210121T1117Z
sb-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29849__20210121T1117Z
sb-pinot-server-1 controllerserver [Times: user=0.04 sys=0.00, real=0.01 secs]
sb-pinot-controller-1 controller2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29850__20210121T1117Z
2021-01-21T17:25:04.137+0000: 1350.530: [GC concurrent-root-region-scan-start]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 sb-pinot-server-0 controller 2021-01-21T17:25:04.137+0000: 1350.530: Total time for which application threads were stopped: 0.0116309 seconds, Stopping threads took: 0.0001002 seconds
server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
sb-pinot-server-0 sb-pinot-server-1sb-pinot-controller-1 controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-root-region-scan-end, 0.0031610 secs]
server 2021/01/21 17:22:51.966 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29851__20210121T1117Z
sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.966 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29852__20210121T1118Z
sb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29853__20210121T1118Z
server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-server-1controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-mark-start]
sb-pinot-server-0 server [GC Worker Total (ms): Min: 28.4, Avg: 28.4, Max: 28.5, Diff: 0.1, Sum: 170.6]
sb-pinot-server-0 server sb-pinot-controller-1 controller server 2021-01-21T17:25:04.256+0000: 1350.649: Application time: 0.1189493 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.256+0000: 1350.649: [GC pause (G1 Evacuation Pause) (young), 0.0236359 secs]
sb-pinot-controller-1 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29854__20210121T1118Z
controller [Parallel Time: 23.0 ms, GC Workers: 4]
sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350649.5, Avg: 1350649.5, Max: 1350649.5, Diff: 0.1]
[GC Worker End (ms): Min: 168866.4, Avg: 168866.4, Max: 168866.4, Diff: 0.0]
sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29855__20210121T1118Z
[Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.2, Diff: 0.2, Sum: 4.2]
sb-pinot-controller-1 controller [Update RS (ms): Min: 6.5, Avg: 6.6, Max: 6.6, Diff: 0.1, Sum: 26.2]
sb-pinot-controller-1 controller [Processed Buffers: Min: 65, Avg: 70.0, Max: 74, Diff: 9, Sum: 280]
sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Object Copy (ms): Min: 15.2, Avg: 15.2, Max: 15.3, Diff: 0.2, Sum: 61.0]
sb-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-controller-1 controllersb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29856__20210121T1118Z
sb-pinot-server-0 sb-pinot-server-1server [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 22.9, Avg: 22.9, Max: 22.9, Diff: 0.0, Sum: 91.6]
server server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29857__20210121T1118Z
sb-pinot-server-1 [Code Root Purge: 0.0 ms]
sb-pinot-server-0 server [Clear CT: 0.4 ms]
sb-pinot-server-0 server [Other: 3.2 ms]
sb-pinot-server-0 server [Choose CSet: 0.4 ms]
sb-pinot-server-0 server [Ref Proc: 0.8 ms]
sb-pinot-server-0 server [Ref Enq: 0.0 ms]
sb-pinot-server-0 server [Redirty Cards: 0.1 ms]
sb-pinot-server-0 server [Humongous Register: 0.1 ms]
sb-pinot-server-0 server [Humongous Reclaim: 0.1 ms]
sb-pinot-server-0 server [Free CSet: 1.2 ms]
sb-pinot-server-0 server [Eden: 195.0M(195.0M)->0.0B(192.0M) Survivors: 9216.0K->12288.0K Heap: 2064.8M(4096.0M)->1443.5M(4096.0M)]
sb-pinot-server-0 server [Times: user=0.17 sys=0.00, real=0.03 secs]
sb-pinot-server-0 server 2021-01-21T17:25:07.665+0000: 168.870: Total time for which application threads were stopped: 0.0330694 seconds, Stopping threads took: 0.0002922 seconds
sb-pinot-server-0 server 2021-01-21T17:25:07.829+0000: 169.034: Application time: 0.1644559 seconds
sb-pinot-server-0 server 2021-01-21T17:25:07.838+0000: 169.043: [GC pause (G1 Evacuation Pause) (mixed), 0.0260221 secs]
sb-pinot-server-0 server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29858__20210121T1118Z
sb-pinot-server-1sb-pinot-controller-1 server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29859__20210121T1119Z
sb-pinot-server-1 controller [GC Worker End (ms): Min: 1350672.4, Avg: 1350672.4, Max: 1350672.4, Diff: 0.0]
sb-pinot-controller-1 controller [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controller server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29860__20210121T1119Z
sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29861__20210121T1119Z
[Code Root Purge: 0.0 ms]
sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29862__20210121T1119Z
sb-pinot-server-1 sb-pinot-controller-1 controller [Clear CT: 0.1 ms]
sb-pinot-controller-1server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29863__20210121T1119Z
sb-pinot-server-1 server server [Parallel Time: 23.6 ms, GC Workers: 6]
sb-pinot-server-0 server [GC Worker Start (ms): Min: 169043.2, Avg: 169043.2, Max: 169043.3, Diff: 0.1]
sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.5, Max: 1.6, Diff: 0.2, Sum: 8.9]
sb-pinot-server-0 server [Update RS (ms): Min: 13.8, Avg: 13.8, Max: 13.8, Diff: 0.1, Sum: 82.8]
sb-pinot-server-0 server [Processed Buffers: Min: 50, Avg: 54.3, Max: 58, Diff: 8, Sum: 326]
sb-pinot-server-0 server [Scan RS (ms): Min: 0.2, Avg: 0.3, Max: 0.3, Diff: 0.1, Sum: 1.6]
2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29864__20210121T1119Z
sb-pinot-server-1sb-pinot-server-0 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29865__20210121T1119Z
sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29866__20210121T1120Z
sb-pinot-server-1 server controller2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29867__20210121T1120Z
sb-pinot-server-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Object Copy (ms): Min: 7.7, Avg: 7.9, Max: 8.0, Diff: 0.2, Sum: 47.2]
sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.2]
sb-pinot-server-0 server [GC Worker Total (ms): Min: 23.4, Avg: 23.4, Max: 23.5, Diff: 0.1, Sum: 140.7]
[Other: 0.6 ms]
sb-pinot-controller-1 server sb-pinot-server-0 server controller [Choose CSet: 0.0 ms]
sb-pinot-controller-1 controller [Ref Proc: 0.1 ms]
[GC Worker End (ms): Min: 169066.7, Avg: 169066.7, Max: 169066.7, Diff: 0.0]
sb-pinot-server-0 2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29868__20210121T1120Z
sb-pinot-server-1 server 2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29869__20210121T1120Z
server sb-pinot-server-1 [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 server [Code Root Purge: 0.0 ms]
sb-pinot-server-0 server server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29870__20210121T1120Z
sb-pinot-server-1 sb-pinot-controller-1 controller [Ref Enq: 0.0 ms]
[Clear CT: 0.2 ms]
sb-pinot-controller-1 controller sb-pinot-server-0 server [Redirty Cards: 0.1 ms]
sb-pinot-controller-1 [Other: 2.2 ms]
sb-pinot-server-0 controller server [Choose CSet: 0.0 ms]
[Humongous Register: 0.0 ms]
sb-pinot-server-0 server [Ref Proc: 1.0 ms]
sb-pinot-server-0 server [Ref Enq: 0.0 ms]
sb-pinot-server-0 server [Redirty Cards: 0.1 ms]
sb-pinot-server-0 sb-pinot-controller-1 controller server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29871__20210121T1120Z
sb-pinot-server-1 server [Humongous Register: 0.2 ms]
sb-pinot-server-0 server [Humongous Reclaim: 0.1 ms]
sb-pinot-server-0 server [Free CSet: 0.3 ms]
server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29872__20210121T1120Z
sb-pinot-server-1 server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29873__20210121T1120Z
sb-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29874__20210121T1120Z
sb-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29875__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.974 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29876__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29877__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29878__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29879__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29880__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29881__20210121T1121Z
sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29882__20210121T1121Z
sb-pinot-server-1 [Humongous Reclaim: 0.1 ms]
sb-pinot-controller-1 controller [Free CSet: 0.1 ms]
sb-pinot-controller-1 controller [Eden: 152.0M(152.0M)->0.0B(298.0M) Survivors: 3072.0K->20480.0K Heap: 638.1M(1024.0M)->482.3M(1024.0M)]
sb-pinot-controller-1 controller [Times: user=0.09 sys=0.00, real=0.03 secs]
sb-pinot-controller-1 controller 2021-01-21T17:25:04.280+0000: 1350.673: Total time for which application threads were stopped: 0.0241836 seconds, Stopping threads took: 0.0000443 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.955: [GC concurrent-mark-end, 0.4221306 secs]
sb-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29883__20210121T1122Z
sb-pinot-server-0 sb-pinot-server-1server [Eden: 192.0M(192.0M)->0.0B(2155.0M) Survivors: 12288.0K->23552.0K Heap: 1650.1M(4096.0M)->1449.3M(4096.0M)]
sb-pinot-server-0 controllerserver [Times: user=0.14 sys=0.00, real=0.03 secs]
sb-pinot-server-0 server 2021-01-21T17:25:07.864+0000: 169.069: Total time for which application threads were stopped: 0.0349989 seconds, Stopping threads took: 0.0082034 seconds
sb-pinot-server-0 server 2021-01-21T17:25:09.808+0000: 171.013: Application time: 1.9435757 seconds
sb-pinot-server-0 serverserver 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29884__20210121T1122Z
sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29885__20210121T1122Z
2021-01-21T17:25:09.813+0000: 171.018: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0428457 secs]
sb-pinot-server-0 2021-01-21T17:25:04.562+0000: 1350.955: Application time: 0.2822203 seconds
server [Parallel Time: 37.6 ms, GC Workers: 6]
sb-pinot-server-0 sb-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.956: [GC remark 2021-01-21T17:25:04.562+0000: 1350.956: [Finalize Marking, 0.0002598 secs] 2021-01-21T17:25:04.563+0000: 1350.956: [GC ref-proc, 0.0004448 secs] 2021-01-21T17:25:04.563+0000: 1350.957: [Unloading, 0.0095395 secs], 0.0108947 secs]
sb-pinot-controller-1 sb-pinot-server-1 controller [Times: user=0.04 sys=0.00, real=0.01 secs]
sb-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Total time for which application threads were stopped: 0.0114314 seconds, Stopping threads took: 0.0000926 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Application time: 0.0000628 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.574+0000: 1350.967: [GC cleanup 774M->655M(1024M), 0.0018691 secs]
sb-pinot-controller-1 controller [Times: user=0.01 sys=0.00, real=0.00 secs]
sb-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29886__20210121T1122Z
controller sb-pinot-server-1 server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29887__20210121T1122Z
server [GC Worker Start (ms): Min: 171018.3, Avg: 171018.4, Max: 171018.4, Diff: 0.1]
sb-pinot-server-0 sb-pinot-server-1server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.4, Max: 1.6, Diff: 0.2, Sum: 8.6]
sb-pinot-server-0 server [Update RS (ms): Min: 20.6, Avg: 20.7, Max: 20.8, Diff: 0.1, Sum: 124.3]
sb-pinot-server-0 server [Processed Buffers: Min: 144, Avg: 149.8, Max: 159, Diff: 15, Sum: 899]
sb-pinot-server-0 2021-01-21T17:25:04.576+0000: 1350.969: Total time for which application threads were stopped: 0.0023571 seconds, Stopping threads took: 0.0000678 seconds
sb-pinot-controller-1 server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29888__20210121T1122Z
sb-pinot-server-1 controllerserver [Scan RS (ms): Min: 0.4, Avg: 0.4, Max: 0.5, Diff: 0.1, Sum: 2.6]
server 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-start]
sb-pinot-controller-1 sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-server-0 server [Object Copy (ms): Min: 14.5, Avg: 14.7, Max: 14.8, Diff: 0.2, Sum: 88.2]
sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
sb-pinot-server-0 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29889__20210121T1122Z
sb-pinot-server-1 controller 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-end, 0.0001739 secs]
server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29890__20210121T1123Z
sb-pinot-controller-1 server [GC Worker Total (ms): Min: 37.3, Avg: 37.4, Max: 37.4, Diff: 0.1, Sum: 224.2]
sb-pinot-server-0 server [GC Worker End (ms): Min: 171055.7, Avg: 171055.7, Max: 171055.8, Diff: 0.1]
sb-pinot-server-0 server [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 server controller 2021-01-21T17:25:04.654+0000: 1351.048: Application time: 0.0785958 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.655+0000: 1351.048: [GC pause (G1 Evacuation Pause) (young), 0.0666046 secs]
sb-pinot-server-1 [Code Root Purge: 0.0 ms]
server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29891__20210121T1123Z
sb-pinot-server-1 sb-pinot-controller-1 controller [Parallel Time: 65.5 ms, GC Workers: 4]
sb-pinot-controller-1 sb-pinot-server-0 server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29892__20210121T1123Z
server [Clear CT: 0.5 ms]
sb-pinot-server-0 server [Other: 4.8 ms]
sb-pinot-server-0 server [Choose CSet: 0.0 ms]
sb-pinot-server-0 server [Ref Proc: 2.2 ms]
sb-pinot-server-0 server [Ref Enq: 0.0 ms]
sb-pinot-server-0 server [Redirty Cards: 0.1 ms]
sb-pinot-server-0 server [Humongous Register: 0.1 ms]
sb-pinot-server-0 server [Humongous Reclaim: 0.6 ms]
controller sb-pinot-server-0 [GC Worker Start (ms): Min: 1351048.4, Avg: 1351048.4, Max: 1351048.4, Diff: 0.0]
server [Free CSet: 1.0 ms]
sb-pinot-controller-1 sb-pinot-server-1 sb-pinot-server-0 server [Eden: 1386.0M(2155.0M)->0.0B(2134.0M) Survivors: 23552.0K->48128.0K Heap: 3188.4M(4096.0M)->1474.4M(4096.0M)]
sb-pinot-server-0 server [Times: user=0.23 sys=0.00, real=0.04 secs]
sb-pinot-server-0 server controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.6]
sb-pinot-controller-1 controller server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29893__20210121T1123Z
2021/01/21 17:25:10.641 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=sblog__0__13120__20210119T1725Z, expected=ONLINE, found=OFFLINE, creationTime=1611249776061, modifiedTime=1611249910406, version=1030, waitingFor=CurrentStateMatch, resource=sblog_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
sb-pinot-server-0 [Update RS (ms): Min: 16.0, Avg: 16.0, Max: 16.0, Diff: 0.1, Sum: 63.9]
server 2021-01-21T17:25:09.856+0000: 2021-01-21T17:25:09.856+0000: 171.061: 171.061: [GC concurrent-root-region-scan-start]
sb-pinot-server-0 server sb-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 111.0, Max: 117, Diff: 14, Sum: 444]
sb-pinot-server-1 server Total time for which application threads were stopped: 0.0480709 seconds, Stopping threads took: 0.0044806 seconds
sb-pinot-server-0 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-root-region-scan-end, 0.0198202 secs]
2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29894__20210121T1123Z
sb-pinot-server-1 sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.1, Diff: 0.0, Sum: 0.3]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1server 2021/01/21 17:22:51.980 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29895__20210121T1123Z
sb-pinot-server-1 server 2021/01/21 17:22:51.980 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29896__20210121T1123Z
sb-pinot-server-1 controller [Object Copy (ms): Min: 48.1, Avg: 48.2, Max: 48.3, Diff: 0.2, Sum: 192.8]
server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29897__20210121T1124Z
sb-pinot-server-1 sb-pinot-server-0serversb-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 65.4, Avg: 65.4, Max: 65.4, Diff: 0.0, Sum: 261.7]
sb-pinot-controller-1 controller [GC Worker End (ms): Min: 1351113.8, Avg: 1351113.8, Max: 1351113.8, Diff: 0.0]
sb-pinot-controller-1 controller [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
sb-pinot-controller-1 controller [Clear CT: 0.1 ms]
sb-pinot-controller-1 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-mark-start]
sb-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: [GC concurrent-mark-end, 1.3475822 secs]
2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29898__20210121T1124Z
sb-pinot-server-1 sb-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: Application time: 1.3674839 seconds
server sb-pinot-server-0 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29899__20210121T1124Z
sb-pinot-server-1 controller [Other: 1.0 ms]
sb-pinot-controller-1 controllerserver 2021-01-21T17:25:11.228+0000: 172.433: [GC remark 2021-01-21T17:25:11.228+0000: 172.433: [Finalize Marking, 0.0003440 secs] 2021-01-21T17:25:11.229+0000: 172.434: [GC ref-proc, 0.0850389 secs] 2021-01-21T17:25:11.314+0000: 172.519: [Unloading, 0.0079844 secs], 0.0957425 secs]
sb-pinot-server-0 server [Times: user=0.11 sys=0.03, real=0.10 secs]
sb-pinot-server-0 server 2021-01-21T17:25:11.324+0000: 172.529: Total time for which application threads were stopped: 0.1009431 seconds, Stopping threads took: 0.0045995 seconds
server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29900__20210121T1124Z
[Choose CSet: 0.0 ms]
sb-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29901__20210121T1124Z
sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29902__20210121T1124Z
sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29903__20210121T1125Z
sb-pinot-server-1 controller2021-01-21T17:25:11.324+0000: 172.529: Application time: 0.0000758 seconds
sb-pinot-server-0 server server 2021-01-21T17:25:11.325+0000: 172.530: [GC cleanup 2780M->2780M(4096M), 0.0035921 secs]
sb-pinot-server-0 [Ref Proc: 0.2 ms]
sb-pinot-controller-1 controller server [Ref Enq: 0.0 ms]
sb-pinot-controller-1 controller [Redirty Cards: 0.3 ms]
[Times: user=0.02 sys=0.00, real=0.00 secs]
sb-pinot-controller-1 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29904__20210121T1125Z
sb-pinot-server-1 serversb-pinot-server-0 server 2021-01-21T17:25:11.328+0000: 172.533: Total time for which application threads were stopped: 0.0041608 seconds, Stopping threads took: 0.0000167 seconds
sb-pinot-server-0 server 2021/01/21 17:22:51.982 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29905__20210121T1125Z
sb-pinot-server-1 2021-01-21T17:25:11.481+0000: 172.686: Application time: 0.1529441 seconds
servercontrollersb-pinot-server-0 server 2021-01-21T17:25:11.482+0000: 172.687: Total time for which application threads were stopped: 0.0005239 seconds, Stopping threads took: 0.0000613 seconds
sb-pinot-server-0 [Humongous Register: 0.0 ms]
serversb-pinot-controller-1 2021/01/21 17:22:51.982 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29906__20210121T1125Z
sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29907__20210121T1125Z
sb-pinot-server-1 controller server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29908__20210121T1125Z
sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29909__20210121T1125Z
[Humongous Reclaim: 0.1 ms]
sb-pinot-controller-1sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29910__20210121T1125Z
sb-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29911__20210121T1126Z
controller [Free CSet: 0.2 ms]
sb-pinot-controller-1 controller [Eden: 298.0M(298.0M)->0.0B(11264.0K) Survivors: 20480.0K->40960.0K Heap: 700.6M(1024.0M)->429.5M(1024.0M)]
sb-pinot-controller-1 controller [Times: user=0.26 sys=0.00, real=0.07 secs]
sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29912__20210121T1126Z
sb-pinot-server-1 server 2021-01-21T17:25:11.489+0000: 172.694: Application time: 0.0071873 seconds
sb-pinot-server-0 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29913__20210121T1126Z
sb-pinot-server-1 server 2021-01-21T17:25:11.490+0000: 172.694: Total time for which application threads were stopped: 0.0004430 seconds, Stopping threads took: 0.0000272 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.551+0000: 172.755: Application time: 0.0610437 seconds
2021-01-21T17:25:04.722+0000: 1351.115: Total time for which application threads were stopped: 0.0671829 seconds, Stopping threads took: 0.0000684 seconds
sb-pinot-controller-1 sb-pinot-server-0controller 2021-01-21T17:25:04.737+0000: 1351.130: Application time: 0.0150208 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.737+0000: 1351.130: [GC pause (G1 Evacuation Pause) (mixed), 0.0775762 secs]
server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29914__20210121T1126Z
server 2021-01-21T17:25:11.552+0000: 172.757: Total time for which application threads were stopped: 0.0013581 seconds, Stopping threads took: 0.0000241 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.710+0000: 172.915: Application time: 0.1581764 seconds
sb-pinot-server-0 sb-pinot-server-1 server sb-pinot-controller-1 controller [Parallel Time: 76.1 ms, GC Workers: 4]
sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1351130.8, Avg: 1351130.8, Max: 1351130.8, Diff: 0.1]
sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.2]
sb-pinot-controller-1 controller [Update RS (ms): Min: 31.3, Avg: 31.4, Max: 31.4, Diff: 0.1, Sum: 125.4]
sb-pinot-controller-1 controller [Processed Buffers: Min: 124, Avg: 125.0, Max: 126, Diff: 2, Sum: 500]
sb-pinot-controller-1 server controller 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29915__20210121T1126Z
sb-pinot-server-1 2021-01-21T17:25:11.718+0000: 172.923: Total time for which application threads were stopped: 0.0080070 seconds, Stopping threads took: 0.0074375 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.724+0000: 172.928: Application time: 0.0054288 seconds
sb-pinot-server-0server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29916__20210121T1126Z
sb-pinot-server-1 server 2021/01/21 17:22:51.986 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29917__20210121T1126Z
sb-pinot-server-1 [Scan RS (ms): Min: 13.4, Avg: 13.5, Max: 13.7, Diff: 0.3, Sum: 54.1]
sb-pinot-controller-1 controllerserver [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5]
2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29918__20210121T1127Z
sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29919__20210121T1127Z
controller [Object Copy (ms): Min: 29.8, Avg: 29.9, Max: 30.1, Diff: 0.3, Sum: 119.8]
sb-pinot-controller-1 server controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29920__20210121T1127Z
sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29921__20210121T1127Z
2021-01-21T17:25:11.728+0000: 172.933: Total time for which application threads were stopped: 0.0044722 seconds, Stopping threads took: 0.0031338 seconds
sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29922__20210121T1127Z
sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29923__20210121T1127Z
sb-pinot-controller-1 sb-pinot-server-0 server controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
sb-pinot-controller-1 2021-01-21T17:25:11.732+0000: 172.937: Application time: 0.0044179 seconds
sb-pinot-server-0 controllerserver sb-pinot-server-1 [GC Worker Total (ms): Min: 76.0, Avg: 76.0, Max: 76.0, Diff: 0.0, Sum: 304.1]
sb-pinot-controller-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29924__20210121T1127Z
sb-pinot-server-1 controller [GC Worker End (ms): Min: 1351206.8, Avg: 1351206.8, Max: 1351206.8, Diff: 0.0]
server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29925__20210121T1127Z
sb-pinot-server-1 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0004491 seconds, Stopping threads took: 0.0000215 seconds
sb-pinot-server-0 server sb-pinot-controller-1 controller 2021-01-21T17:25:11.733+0000: 172.938: Application time: 0.0000363 seconds
sb-pinot-server-0 [Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 controllerserver 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0003160 seconds, Stopping threads took: 0.0000079 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Application time: 0.0124480 seconds
[Code Root Purge: 0.0 ms]
sb-pinot-controller-1 server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29926__20210121T1128Z
sb-pinot-server-1 controller [Clear CT: 0.1 ms]
server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29927__20210121T1128Z
sb-pinot-server-1 server sb-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Total time for which application threads were stopped: 0.0003742 seconds, Stopping threads took: 0.0000195 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.759+0000: 172.964: Application time: 0.0131737 seconds
sb-pinot-server-02021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29928__20210121T1128Z
sb-pinot-server-1 server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29929__20210121T1128Z
sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29930__20210121T1128Z
sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29931__20210121T1128Z
sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29932__20210121T1128Z
sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29933__20210121T1129Z
sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29934__20210121T1129Z
server sb-pinot-server-1 controller [Other: 1.3 ms]
sb-pinot-controller-1 controller [Choose CSet: 0.2 ms]
sb-pinot-controller-1 controller [Ref Proc: 0.1 ms]
sb-pinot-controller-1 controller [Ref Enq: 0.0 ms]
sb-pinot-controller-1 2021-01-21T17:25:11.760+0000: 172.965: Total time for which application threads were stopped: 0.0007156 seconds, Stopping threads took: 0.0002863 seconds
controller [Redirty Cards: 0.4 ms]
sb-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Application time: 0.0087255 seconds
sb-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Total time for which application threads were stopped: 0.0007774 seconds, Stopping threads took: 0.0002464 seconds
sb-pinot-controller-1 controller [Humongous Register: 0.0 ms]
sb-pinot-controller-1 sb-pinot-server-0 server 2021-01-21T17:25:11.771+0000: 172.976: Application time: 0.0015455 seconds
controller sb-pinot-server-0 server 2021-01-21T17:25:11.773+0000: 172.977: Total time for which application threads were stopped: 0.0015423 seconds, Stopping threads took: 0.0002360 seconds
sb-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: Application time: 0.5611696 seconds
sb-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: [GC pause (G1 Evacuation Pause) (young), 0.2155960 secs]
sb-pinot-server-0 [Humongous Reclaim: 0.0 ms]
serversb-pinot-controller-1 controller [Free CSet: 0.4 ms]
sb-pinot-controller-1 controller [Eden: 11264.0K(11264.0K)->0.0B(48128.0K) Survivors: 40960.0K->4096.0K Heap: 455.0M(1024.0M)->354.2M(1024.0M)]
sb-pinot-controller-1 [Parallel Time: 208.2 ms, GC Workers: 6]
sb-pinot-server-0 server server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29935__20210121T1129Z
[GC Worker Start (ms): Min: 173539.5, Avg: 173539.5, Max: 173539.5, Diff: 0.1]
sb-pinot-server-1 controller server sb-pinot-server-0 [Times: user=0.30 sys=0.00, real=0.07 secs]
server [Ext Root Scanning (ms): Min: 1.1, Avg: 1.3, Max: 1.6, Diff: 0.4, Sum: 7.7]
2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29936__20210121T1129Z
sb-pinot-server-1 sb-pinot-controller-1 controller 2021-01-21T17:25:04.815+0000: 1351.208: Total time for which application threads were stopped: 0.0781603 seconds, Stopping threads took: 0.0000425 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.841+0000: 1351.235: Application time: 0.0263536 seconds
sb-pinot-controller-1 controller 2021-01-21T17:25:04.842+0000: 1351.235: [GC pause (G1 Evacuation Pause) (mixed), 0.0820991 secs]
server sb-pinot-controller-1 controller 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29937__20210121T1129Z
sb-pinot-server-1 [Parallel Time: 81.1 ms, GC Workers: 4]
sb-pinot-server-0 sb-pinot-controller-1 server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29938__20210121T1129Z
controller [GC Worker Start (ms): Min: 1351235.2, Avg: 1351235.2, Max: 1351235.2, Diff: 0.0]
sb-pinot-controller-1 servercontroller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.3]
sb-pinot-controller-1 [Update RS (ms): Min: 15.5, Avg: 15.5, Max: 15.6, Diff: 0.1, Sum: 93.3]
sb-pinot-server-0 server [Processed Buffers: Min: 108, Avg: 112.2, Max: 115, Diff: 7, Sum: 673]
sb-pinot-server-0 server [Scan RS (ms): Min: 166.2, Avg: 166.3, Max: 166.5, Diff: 0.2, Sum: 998.0]
sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29939__20210121T1129Z
sb-pinot-server-1 controller server [Update RS (ms): Min: 35.6, Avg: 35.7, Max: 35.9, Diff: 0.4, Sum: 142.9]
sb-pinot-controller-1 controller [Processed Buffers: Min: 150, Avg: 155.0, Max: 159, Diff: 9, Sum: 620]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
sb-pinot-controller-1 controller [Scan RS (ms): Min: 5.9, Avg: 14.1, Max: 22.5, Diff: 16.6, Sum: 56.4]
sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 4.2, Max: 16.4, Diff: 16.4, Sum: 16.8]
server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29940__20210121T1129Z
sb-pinot-server-1 server 2021/01/21 17:22:51.991 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29941__20210121T1129Z
sb-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29942__20210121T1129Z
sb-pinot-controller-1 sb-pinot-server-1 controller sb-pinot-server-0 [Object Copy (ms): Min: 21.7, Avg: 25.9, Max: 38.2, Diff: 16.6, Sum: 103.5]
server server sb-pinot-controller-1 controller 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29943__20210121T1130Z
sb-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29944__20210121T1130Z
sb-pinot-server-1 [Object Copy (ms): Min: 24.4, Avg: 24.8, Max: 25.0, Diff: 0.6, Sum: 148.7]
sb-pinot-server-0 server server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
sb-pinot-server-02021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29945__20210121T1130Z
sb-pinot-server-1 server [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 7]
sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server [GC Worker Total (ms): Min: 208.0, Avg: 208.0, Max: 208.1, Diff: 0.1, Sum: 1248.2]
2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29946__20210121T1130Z
sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29947__20210121T1130Z
sb-pinot-server-1 sb-pinot-server-0 server [GC Worker End (ms): Min: 173747.5, Avg: 173747.5, Max: 173747.6, Diff: 0.1]
server sb-pinot-server-0 server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29948__20210121T1130Z
sb-pinot-server-1 controller [Termination Attempts: Min: 1, Avg: 5.8, Max: 10, Diff: 9, Sum: 23]
sb-pinot-controller-1 [Code Root Fixup: 0.0 ms]
sb-pinot-server-0 server controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29949__20210121T1130Z
sb-pinot-server-1 server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29950__20210121T1130Z
sb-pinot-server-1 server 2021/01/21 17:22:51.995 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29951__20210121T1130Z
sb-pinot-server-1 server sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 81.0, Avg: 81.0, Max: 81.0, Diff: 0.0, Sum: 324.0]
[Code Root Purge: 0.0 ms]
2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29952__20210121T1130Z
sb-pinot-server-1 sb-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29953__20210121T1130Z
sb-pinot-controller-1 controller [GC Worker End (ms): Min: 1351316.2, Avg: 1351316.2, Max: 1351316.2, Diff: 0.0]
sb-pinot-controller-1 [Clear CT: 0.9 ms]
controller sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29954__20210121T1130Z
sb-pinot-server-0 sb-pinot-server-1 server [Other: 6.5 ms]
[Code Root Fixup: 0.0 ms]
sb-pinot-controller-1 sb-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29955__20210121T1130Z
[Choose CSet: 0.0 ms]
sb-pinot-server-1 server sb-pinot-server-0 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29956__20210121T1131Z
server [Ref Proc: 3.8 ms]
sb-pinot-server-0 server [Ref Enq: 0.0 ms]
sb-pinot-server-0 server [Redirty Cards: 0.2 ms]
sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29957__20210121T1131Z
sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29958__20210121T1131Z
sb-pinot-server-1 controller [Code Root Purge: 0.0 ms]
sb-pinot-controller-1 controller [Clear CT: 0.1 ms]
server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29959__20210121T1131Z
server sb-pinot-server-1 server [Humongous Register: 0.2 ms]
sb-pinot-server-0 sb-pinot-controller-1 controller 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29960__20210121T1131Z
sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29961__20210121T1131Z
sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29962__20210121T1132Z
sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29963__20210121T1132Z
sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29964__20210121T1132Z
[Other: 0.8 ms]
server [Humongous Reclaim: 0.7 ms]
sb-pinot-server-0 sb-pinot-controller-1 server [Free CSet: 1.4 ms]
controller sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29965__20210121T1132Z
[Choose CSet: 0.1 ms]
sb-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29966__20210121T1132Z
sb-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29967__20210121T1132Z
sb-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29968__20210121T1132Z
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764481860
> @fx19880617 , Max open file limit was unlimited however I had to update vm max map count in os. After that I restarted Pinot cluster and this came right after the restart of Pinot Server. But it disappeared after 10 mts and cluster recovered automatically. It is good to know that there is a periodical check to fix the segments. Most of these segments in the ERROR list are completed ones. Do you think I can consider the Cluster healthy then?
If all segments are in ONLINE/CONSUMING status, then you are good.
So the fix is not on the k8s side but on VM OS config.
@daniellavoie just FYI ^^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765281216
> @fx19880617 do you mean the controller data folder? I have mapped /var/pinot/controller/data to a SSD and its max size is 10GB however it was only 50% full. I am using 2 controller and both were just around 50% full.
>
> I have some questions:
> a) Is it ok if I move this controller data folder to EFS? So it will grow automatically. Or will it cause any latency?. Please note for pinot server I will continue using SSD for performance. I am planning to use EFS just for controller data.
>
> b) I have to create small segments because the textindex which I am relying heavily. Textindex have an issue where it only scans online or completed segments(#6431) . Because of this behavior I am creating small segments so that I can get instant results. Is there any way I can merge completed segments with some periodic process?
>
> c) Once I restarted controller, broker and server at the same time the issue went away and ingestion started working. However individual restart did not helped. Why this is so?
a) For controller storage, it's only on data load/backup path, not query path, so it's recommended to use blob store like s3/hdfs.
b) I think we can leverage minion framework to generate tasks to merge small segments, this task is not yet implemented, which we can implement.
c) Do you mean restart server only won't fix? Typically restart servers will fix the error segments issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764822003
These are my Cluster info
Number of Segments : 23114 / 23115
Status : BAD (in the UI ) This is a bug?
Estimated Size : 84312409908
Reported Size: 84312409908
To Start the ingestion back I tried to restart controller, broker , pinot server one by one to avoid outage. However that didn't work. So i have to start all controllers, brokers and servers at once. After that ingestion resumed and caught up with kafka.
Not sure what is going on with the cluster..!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
@fx19880617 Thanks for the details. I just keep monitoring the system and the memory for the System was going through the edge. I noticed that once it happens and If i leave it for say couple of days following errors will appear and Pinot server will stop ingestion completely. Is there any way to fix this issue?
```
my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.5, Avg: 1.6, Max: 1.8, Diff: 0.3, Sum: 9.4]
2021/01/21 17:02:38.123 ERROR [ServerSegmentCompletionProtocolHandler] [mylog__0__32545__20210121T1702Z] Could not send request http://my-pinot-controller-0.my-pinot-controller-headless.logging.svc.cluster.local:9000/segmentCommit?name=mylog__0__32545__20210121T1702Z&offset=44504082&instance=Server_my-pinot-server-1.my-pinot-server-headless.logging.svc.cluster.local_8098&buildTimeMillis=191&memoryUsedBytes=256504&segmentSizeBytes=2763568&rowCount=1000&streamPartitionMsgOffset=44504082
my-pinot-server-1 my-pinot-server-0server server [Update RS (ms): Min: 5.6, Avg: 5.7, Max: 5.7, Diff: 0.1, Sum: 34.1]
my-pinot-controller-1 shaded.org.apache.http.NoHttpResponseException: my-pinot-controller-0.my-pinot-controller-headless.logging.svc.cluster.local:9000 failed to respond
my-pinot-server-1my-pinot-server-0 my-pinot-broker-0 server [Processed Buffers: Min: 17, Avg: 27.3, Max: 34, Diff: 17, Sum: 164]
at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-a36d1c04c720c0f47dfec078c24c84c98ed665c4]
my-pinot-broker-0 my-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
broker [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-1 my-pinot-broker-0 server broker [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 2]
my-pinot-broker-0 my-pinot-controller-1 server [Choose CSet: 0.0 ms]
my-pinot-server-0 broker controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
... 16 more
my-pinot-controller-1server [Ref Proc: 3.4 ms]
controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-server-1 server 2021/01/21 17:04:50.874 ERROR [LLRealtimeSegmentDataManager_mylog__0__32545__20210121T1702Z] [mylog__0__32545__20210121T1702Z] Holding after response from Controller: {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"NOT_SENT","offset":-1}
my-pinot-server-1 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
server my-pinot-server-0 server [Ref Enq: 0.0 ms]
my-pinot-server-1 controller 2021-01-21T17:25:03.929+0000: 1350.323: Application time: 0.0590331 seconds
my-pinot-server-0 my-pinot-controller-1 server controller 2021-01-21T17:25:03.930+0000: 1350.323: [GC pause (G1 Evacuation Pause) (mixed), 0.0406644 secs]
server [Times: user=0.26 sys=0.00, real=0.05 secs]
my-pinot-server-0 2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29726__20210121T1102Z
server 2021-01-21T17:25:05.778+0000: 166.983: Total time for which application threads were stopped: 0.0500843 seconds, Stopping threads took: 0.0000282 seconds
my-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: [GC concurrent-mark-end, 1.4470120 secs]
my-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: Application time: 0.4400702 seconds
my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29727__20210121T1102Z
my-pinot-server-1 server 2021-01-21T17:25:06.219+0000: 167.424: [GC remark 2021-01-21T17:25:06.219+0000: 167.424: [Finalize Marking, 0.0002085 secs] 2021-01-21T17:25:06.219+0000: 167.424: [GC ref-proc, 0.0851322 secs] 2021-01-21T17:25:06.304+0000: 167.509: [Unloading, 0.0080895 secs], 0.0963586 secs]
2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29728__20210121T1102Z
my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29729__20210121T1102Z
my-pinot-server-1my-pinot-server-0 server my-pinot-controller-1 [Times: user=0.11 sys=0.04, real=0.10 secs]
my-pinot-server-0server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29730__20210121T1102Z
my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29731__20210121T1102Z
my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29732__20210121T1102Z
controller [Parallel Time: 39.8 ms, GC Workers: 4]
my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350323.3, Avg: 1350323.3, Max: 1350323.3, Diff: 0.0]
server 2021-01-21T17:25:06.315+0000: 167.520: Total time for which application threads were stopped: 0.0972538 seconds, Stopping threads took: 0.0003432 seconds
my-pinot-server-0 server my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 0.9, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.0]
my-pinot-controller-1 my-pinot-server-1 2021-01-21T17:25:06.315+0000: 167.520: Application time: 0.0001148 seconds
servercontroller [Update RS (ms): Min: 15.6, Avg: 15.6, Max: 15.6, Diff: 0.0, Sum: 62.4]
my-pinot-controller-1 my-pinot-server-0 servercontroller [Processed Buffers: Min: 76, Avg: 77.8, Max: 79, Diff: 3, Sum: 311]
my-pinot-controller-1 controller [Scan RS (ms): Min: 1.7, Avg: 1.7, Max: 1.7, Diff: 0.1, Sum: 6.8]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 2021-01-21T17:25:06.316+0000: 167.521: [GC cleanup 3168M->3047M(4096M), 0.0040580 secs]
my-pinot-server-0 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29733__20210121T1102Z
my-pinot-server-1server [Times: user=0.02 sys=0.00, real=0.00 secs]
my-pinot-server-0 server server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29734__20210121T1103Z
controller [Object Copy (ms): Min: 21.3, Avg: 21.4, Max: 21.5, Diff: 0.2, Sum: 85.6]
my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29735__20210121T1103Z
my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29736__20210121T1103Z
controller2021-01-21T17:25:06.320+0000: 167.525: Total time for which application threads were stopped: 0.0046889 seconds, Stopping threads took: 0.0001052 seconds
my-pinot-server-0 my-pinot-server-1server 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-start]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-controller-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29737__20210121T1103Z
controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29738__20210121T1103Z
my-pinot-controller-1 server my-pinot-server-1 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-end, 0.0001522 secs]
my-pinot-server-0server server controller [GC Worker Total (ms): Min: 39.7, Avg: 39.7, Max: 39.7, Diff: 0.0, Sum: 158.9]
2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29739__20210121T1103Z
my-pinot-controller-1 2021-01-21T17:25:06.929+0000: 168.134: Application time: 0.6090475 seconds
my-pinot-server-0 my-pinot-server-1 server controller [GC Worker End (ms): Min: 1350363.0, Avg: 1350363.0, Max: 1350363.0, Diff: 0.0]
my-pinot-controller-1 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29740__20210121T1103Z
my-pinot-server-1 serverserver 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29741__20210121T1103Z
my-pinot-server-1 controller [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controller 2021-01-21T17:25:06.930+0000: 168.135: [GC pause (G1 Evacuation Pause) (young), 0.0357839 secs]
my-pinot-server-0 server [Parallel Time: 32.1 ms, GC Workers: 6]
my-pinot-server-0 server [GC Worker Start (ms): Min: 168134.9, Avg: 168134.9, Max: 168134.9, Diff: 0.1]
[Code Root Purge: 0.0 ms]
my-pinot-controller-1my-pinot-server-0 server controller [Clear CT: 0.1 ms]
my-pinot-controller-1 controller [Other: 0.8 ms]
my-pinot-controller-1 controller server 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29742__20210121T1104Z
my-pinot-server-1 [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.5, Diff: 0.4, Sum: 7.9]
my-pinot-server-0 server [Choose CSet: 0.1 ms]
my-pinot-controller-1 controller [Ref Proc: 0.2 ms]
my-pinot-controller-1 controller [Ref Enq: 0.0 ms]
my-pinot-controller-1 controller server [Update RS (ms): Min: 14.8, Avg: 14.9, Max: 15.1, Diff: 0.3, Sum: 89.2]
my-pinot-server-0 server [Processed Buffers: Min: 107, Avg: 122.2, Max: 132, Diff: 25, Sum: 733]
my-pinot-server-0 server [Scan RS (ms): Min: 1.9, Avg: 2.1, Max: 2.2, Diff: 0.3, Sum: 12.5]
my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Redirty Cards: 0.2 ms]
my-pinot-controller-1 controller 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29743__20210121T1104Z
my-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29744__20210121T1104Z
[Humongous Register: 0.0 ms]
my-pinot-controller-1 my-pinot-server-0 server [Object Copy (ms): Min: 13.5, Avg: 13.7, Max: 13.8, Diff: 0.3, Sum: 82.1]
my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
controller [Humongous Reclaim: 0.0 ms]
my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29745__20210121T1104Z
my-pinot-server-0 my-pinot-server-1 [Free CSet: 0.1 ms]
my-pinot-controller-1 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.3]
servermy-pinot-server-0controller server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29746__20210121T1104Z
my-pinot-server-1 [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 466.6M(1024.0M)->412.2M(1024.0M)]
my-pinot-controller-1 server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29747__20210121T1104Z
my-pinot-server-1 server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29748__20210121T1105Z
[GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.1, Diff: 0.1, Sum: 192.1]
my-pinot-server-1 my-pinot-server-0 server 2021/01/21 17:22:51.947 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29749__20210121T1105Z
my-pinot-server-1 server server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29750__20210121T1105Z
controller [Times: user=0.16 sys=0.00, real=0.04 secs]
[GC Worker End (ms): Min: 168166.9, Avg: 168166.9, Max: 168166.9, Diff: 0.1]
my-pinot-controller-1 my-pinot-server-1 controller 2021-01-21T17:25:03.971+0000: 1350.364: Total time for which application threads were stopped: 0.0412316 seconds, Stopping threads took: 0.0000680 seconds
my-pinot-controller-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29751__20210121T1105Z
my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29752__20210121T1105Z
my-pinot-server-1 servermy-pinot-server-0 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29753__20210121T1105Z
server [Code Root Fixup: 0.0 ms]
my-pinot-server-0 controller 2021-01-21T17:25:04.025+0000: 1350.418: Application time: 0.0544185 seconds
my-pinot-controller-1 controllerserver 2021-01-21T17:25:04.025+0000: 1350.419: [GC pause (G1 Evacuation Pause) (mixed), 0.0437685 secs]
[Code Root Purge: 0.0 ms]
my-pinot-server-0 server [Clear CT: 0.5 ms]
my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29754__20210121T1105Z
my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29755__20210121T1105Z
my-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29756__20210121T1106Z
my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29757__20210121T1106Z
server [Other: 3.2 ms]
my-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29758__20210121T1106Z
my-pinot-server-1 my-pinot-server-0 controller server [Choose CSet: 0.0 ms]
[Parallel Time: 43.0 ms, GC Workers: 4]
my-pinot-server-0 server [Ref Proc: 1.6 ms]
servermy-pinot-server-0 server [Ref Enq: 0.0 ms]
2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29759__20210121T1106Z
my-pinot-server-1 my-pinot-server-0 serverserver [Redirty Cards: 0.1 ms]
my-pinot-controller-1 my-pinot-server-0 controller 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29760__20210121T1106Z
my-pinot-server-1 server 2021/01/21 17:22:51.950 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29761__20210121T1106Z
my-pinot-server-1 [GC Worker Start (ms): Min: 1350418.9, Avg: 1350418.9, Max: 1350418.9, Diff: 0.0]
my-pinot-controller-1 server controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.1]
my-pinot-controller-1 controller [Update RS (ms): Min: 23.6, Avg: 23.6, Max: 23.6, Diff: 0.0, Sum: 94.5]
my-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 106.8, Max: 109, Diff: 6, Sum: 427]
server2021/01/21 17:22:51.950 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29762__20210121T1106Z
my-pinot-server-1 server 2021/01/21 17:22:51.951 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29763__20210121T1107Z
my-pinot-server-1 [Humongous Register: 0.2 ms]
server 2021/01/21 17:22:51.951 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29764__20210121T1107Z
my-pinot-server-1 my-pinot-controller-1 server controller [Scan RS (ms): Min: 0.2, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.8]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29765__20210121T1107Z
controller my-pinot-server-0 my-pinot-server-1 server server [Humongous Reclaim: 0.3 ms]
[Object Copy (ms): Min: 18.0, Avg: 18.1, Max: 18.2, Diff: 0.1, Sum: 72.4]
my-pinot-server-0 my-pinot-controller-1 controller server [Free CSet: 0.8 ms]
my-pinot-server-0 [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29766__20210121T1107Z
my-pinot-server-1 server server2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29767__20210121T1107Z
my-pinot-server-1 controller server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29768__20210121T1107Z
my-pinot-server-1 [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [GC Worker Total (ms): Min: 43.0, Avg: 43.0, Max: 43.0, Diff: 0.0, Sum: 171.9]
server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29769__20210121T1107Z
my-pinot-server-1 [Eden: 900.0M(900.0M)->0.0B(167.0M) Survivors: 32768.0K->37888.0K Heap: 3705.3M(4096.0M)->2630.9M(4096.0M)]
my-pinot-server-0 server my-pinot-controller-1 controller server [GC Worker End (ms): Min: 1350461.9, Avg: 1350461.9, Max: 1350461.9, Diff: 0.0]
my-pinot-controller-1 controller 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29770__20210121T1108Z
my-pinot-server-1 [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
my-pinot-controller-1 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29771__20210121T1108Z
my-pinot-server-1 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29772__20210121T1108Z
[Times: user=0.19 sys=0.00, real=0.03 secs]
my-pinot-server-1my-pinot-server-0 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29773__20210121T1108Z
servermy-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29774__20210121T1108Z
my-pinot-server-1 controllerserver 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29775__20210121T1108Z
[Clear CT: 0.1 ms]
my-pinot-controller-1 controller 2021-01-21T17:25:06.966+0000: 168.170: Total time for which application threads were stopped: 0.0366911 seconds, Stopping threads took: 0.0002408 seconds
my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29776__20210121T1108Z
my-pinot-server-1 server [Other: 0.6 ms]
my-pinot-controller-1 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29777__20210121T1109Z
my-pinot-server-1 controller [Choose CSet: 0.0 ms]
my-pinot-controller-1 server controller 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29778__20210121T1109Z
my-pinot-server-1 2021-01-21T17:25:07.002+0000: 168.207: Application time: 0.0364979 seconds
my-pinot-server-0 server 2021-01-21T17:25:07.004+0000: 168.208: [GC pause (G1 Evacuation Pause) (mixed), 0.0506602 secs]
my-pinot-server-0 server [Parallel Time: 47.3 ms, GC Workers: 6]
[Ref Proc: 0.1 ms]
my-pinot-server-0 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29779__20210121T1109Z
server [GC Worker Start (ms): Min: 168208.9, Avg: 168208.9, Max: 168208.9, Diff: 0.1]
my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.4, Max: 1.6, Diff: 0.3, Sum: 8.3]
my-pinot-server-0 my-pinot-controller-1 controller [Ref Enq: 0.0 ms]
my-pinot-controller-1 controller [Redirty Cards: 0.1 ms]
my-pinot-controller-1 controllermy-pinot-server-1 [Humongous Register: 0.0 ms]
my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
my-pinot-controller-1server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29780__20210121T1109Z
my-pinot-server-1 server server [Update RS (ms): Min: 3.5, Avg: 12.9, Max: 31.6, Diff: 28.1, Sum: 77.6]
my-pinot-server-0 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29781__20210121T1109Z
my-pinot-server-1 controller server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29782__20210121T1109Z
my-pinot-server-1 serverserver [Processed Buffers: Min: 6, Avg: 14.3, Max: 31, Diff: 25, Sum: 86]
2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29783__20210121T1109Z
my-pinot-server-0 server [Scan RS (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: 0.8, Sum: 2.8]
my-pinot-server-1my-pinot-server-0 [Free CSet: 0.1 ms]
my-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 470.8M(1024.0M)->444.5M(1024.0M)]
my-pinot-controller-1 controller [Times: user=0.17 sys=0.00, real=0.05 secs]
my-pinot-controller-1server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 controller server [Object Copy (ms): Min: 14.1, Avg: 32.3, Max: 41.4, Diff: 27.4, Sum: 193.9]
my-pinot-server-0 2021-01-21T17:25:04.069+0000: 1350.463: Total time for which application threads were stopped: 0.0443644 seconds, Stopping threads took: 0.0001010 seconds
my-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29784__20210121T1109Z
my-pinot-server-1server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29785__20210121T1110Z
my-pinot-server-1controller server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29786__20210121T1110Z
my-pinot-server-1 servermy-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29787__20210121T1110Z
my-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29788__20210121T1110Z
my-pinot-server-1 2021-01-21T17:25:04.093+0000: 1350.486: Application time: 0.0234818 seconds
my-pinot-controller-1 [GC Worker Total (ms): Min: 47.1, Avg: 47.1, Max: 47.1, Diff: 0.1, Sum: 282.8]
my-pinot-server-0 server [GC Worker End (ms): Min: 168256.0, Avg: 168256.0, Max: 168256.1, Diff: 0.0]
my-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29789__20210121T1110Z
my-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29790__20210121T1110Z
controller my-pinot-server-1 server 2021-01-21T17:25:04.093+0000: 1350.487: Total time for which application threads were stopped: 0.0004505 seconds, Stopping threads took: 0.0000400 seconds
my-pinot-controller-1 server [Code Root Fixup: 0.0 ms]
my-pinot-server-0 server [Code Root Purge: 0.0 ms]
my-pinot-server-0 server [Clear CT: 0.5 ms]
my-pinot-server-0 server [Other: 2.9 ms]
my-pinot-server-0 server [Choose CSet: 0.2 ms]
my-pinot-server-0 server [Ref Proc: 1.1 ms]
my-pinot-server-02021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29791__20210121T1110Z
my-pinot-server-1 controller 2021-01-21T17:25:04.093+0000: 1350.487: Application time: 0.0000469 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.094+0000: 1350.487: [GC pause (GCLocker Initiated GC) (young), 0.0285995 secs]
my-pinot-controller-1 server controller [Parallel Time: 27.9 ms, GC Workers: 4]
my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350487.1, Avg: 1350487.1, Max: 1350487.1, Diff: 0.0]
2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29792__20210121T1111Z
my-pinot-server-1 server [Ref Enq: 0.0 ms]
my-pinot-server-0 my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.2, Sum: 4.1]
server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29793__20210121T1111Z
my-pinot-server-1 server my-pinot-controller-1 server controller [Update RS (ms): Min: 21.6, Avg: 21.7, Max: 21.9, Diff: 0.2, Sum: 86.8]
2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29794__20210121T1111Z
my-pinot-controller-1 my-pinot-server-1 controller [Processed Buffers: Min: 85, Avg: 88.0, Max: 90, Diff: 5, Sum: 352]
my-pinot-controller-1 [Redirty Cards: 0.1 ms]
controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29795__20210121T1111Z
my-pinot-controller-1 my-pinot-server-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29796__20210121T1111Z
my-pinot-server-1 server 2021/01/21 17:22:51.955 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29797__20210121T1111Z
my-pinot-server-1 server my-pinot-server-0 controllerserver [Humongous Register: 0.1 ms]
2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29798__20210121T1111Z
my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29799__20210121T1111Z
my-pinot-server-1 my-pinot-server-0 server [Object Copy (ms): Min: 4.9, Avg: 5.1, Max: 5.2, Diff: 0.3, Sum: 20.4]
[Humongous Reclaim: 0.1 ms]
my-pinot-controller-1 my-pinot-server-0 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29800__20210121T1111Z
my-pinot-server-1 server server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29801__20210121T1111Z
my-pinot-server-1 server my-pinot-controller-1 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29802__20210121T1112Z
controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-server-1 [Free CSet: 0.9 ms]
my-pinot-server-0 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29803__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29804__20210121T1112Z
my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29805__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29806__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29807__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29808__20210121T1112Z
my-pinot-server-1 server server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29809__20210121T1112Z
controller [GC Worker Total (ms): Min: 27.8, Avg: 27.9, Max: 27.9, Diff: 0.0, Sum: 111.4]
my-pinot-controller-1 [Eden: 167.0M(167.0M)->0.0B(200.0M) Survivors: 37888.0K->4096.0K Heap: 2797.9M(4096.0M)->2222.6M(4096.0M)]
controller [GC Worker End (ms): Min: 1350515.0, Avg: 1350515.0, Max: 1350515.0, Diff: 0.0]
my-pinot-controller-1 my-pinot-server-0 server [Times: user=0.14 sys=0.00, real=0.05 secs]
my-pinot-server-0 controller [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
my-pinot-controller-1 my-pinot-server-1 controller [Clear CT: 0.1 ms]
my-pinot-controller-1 controller [Other: 0.6 ms]
my-pinot-controller-1 controller [Choose CSet: 0.0 ms]
server server 2021-01-21T17:25:07.054+0000: 168.259: Total time for which application threads were stopped: 0.0522104 seconds, Stopping threads took: 0.0008706 seconds
my-pinot-server-0 server 2021-01-21T17:25:07.304+0000: 168.509: Application time: 0.2499433 seconds
my-pinot-server-0 my-pinot-controller-1 server 2021-01-21T17:25:07.305+0000: 168.510: [GC pause (G1 Evacuation Pause) (mixed), 0.0317508 secs]
controller [Ref Proc: 0.1 ms]
my-pinot-controller-1 controller [Ref Enq: 0.0 ms]
my-pinot-server-0 server my-pinot-controller-1 controller [Redirty Cards: 0.1 ms]
[Parallel Time: 29.1 ms, GC Workers: 6]
my-pinot-controller-1 controller 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29810__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29811__20210121T1112Z
[Humongous Register: 0.0 ms]
my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
my-pinot-controller-1 controller my-pinot-server-0 server [GC Worker Start (ms): Min: 168510.7, Avg: 168510.7, Max: 168510.7, Diff: 0.1]
my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.4, Diff: 0.2, Sum: 7.9]
my-pinot-server-1 [Free CSet: 0.1 ms]
my-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(91136.0K) Survivors: 7168.0K->2048.0K Heap: 493.8M(1024.0M)->450.0M(1024.0M)]
my-pinot-server-0 my-pinot-controller-1 server controller [Times: user=0.11 sys=0.00, real=0.03 secs]
my-pinot-controller-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29812__20210121T1112Z
my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29813__20210121T1112Z
controller 2021-01-21T17:25:04.122+0000: 1350.516: Total time for which application threads were stopped: 0.0289920 seconds, Stopping threads took: 0.0000117 seconds
my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29814__20210121T1112Z
my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29815__20210121T1113Z
my-pinot-server-1 [Update RS (ms): Min: 22.7, Avg: 22.7, Max: 22.8, Diff: 0.1, Sum: 136.4]
my-pinot-server-0 server controller server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29816__20210121T1113Z
[Processed Buffers: Min: 90, Avg: 107.3, Max: 116, Diff: 26, Sum: 644]
my-pinot-server-0 server [Scan RS (ms): Min: 1.2, Avg: 1.2, Max: 1.2, Diff: 0.1, Sum: 7.3]
my-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29817__20210121T1113Z
my-pinot-server-1 my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Object Copy (ms): Min: 3.5, Avg: 3.7, Max: 3.8, Diff: 0.2, Sum: 22.1]
server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29818__20210121T1113Z
my-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29819__20210121T1113Z
my-pinot-server-1 2021-01-21T17:25:04.125+0000: 1350.518: Application time: 0.0027123 seconds
my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller server 2021-01-21T17:25:04.125+0000: 1350.519: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0110860 secs]
my-pinot-controller-1 my-pinot-server-0 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29820__20210121T1113Z
server [Termination Attempts: Min: 1, Avg: 1.3, Max: 2, Diff: 1, Sum: 8]
my-pinot-server-0 server my-pinot-server-1 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
controller [Parallel Time: 10.5 ms, GC Workers: 4]
my-pinot-controller-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29821__20210121T1114Z
my-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29822__20210121T1114Z
controller [GC Worker Start (ms): Min: 1350519.1, Avg: 1350519.1, Max: 1350519.1, Diff: 0.0]
my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.2, Max: 1.4, Diff: 0.2, Sum: 5.0]
my-pinot-controller-1 my-pinot-server-1 my-pinot-server-0 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29823__20210121T1114Z
my-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29824__20210121T1114Z
servermy-pinot-server-1controller [Update RS (ms): Min: 8.3, Avg: 8.3, Max: 8.3, Diff: 0.0, Sum: 33.2]
my-pinot-controller-1 controller [GC Worker Total (ms): Min: 28.9, Avg: 29.0, Max: 29.0, Diff: 0.0, Sum: 173.8]
my-pinot-server-0 [Processed Buffers: Min: 37, Avg: 38.8, Max: 41, Diff: 4, Sum: 155]
my-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server [GC Worker End (ms): Min: 168539.7, Avg: 168539.7, Max: 168539.7, Diff: 0.0]
my-pinot-server-0 my-pinot-controller-1 controller [Object Copy (ms): Min: 0.7, Avg: 0.8, Max: 0.9, Diff: 0.2, Sum: 3.3]
my-pinot-controller-1 servercontroller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 5]
my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29825__20210121T1114Z
server [Code Root Fixup: 0.0 ms]
my-pinot-server-0 server [Code Root Purge: 0.0 ms]
my-pinot-server-0 [GC Worker Total (ms): Min: 10.4, Avg: 10.4, Max: 10.4, Diff: 0.0, Sum: 41.5]
my-pinot-controller-1 my-pinot-server-1 server controller [GC Worker End (ms): Min: 1350529.5, Avg: 1350529.5, Max: 1350529.5, Diff: 0.0]
server [Clear CT: 0.4 ms]
2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29826__20210121T1114Z
my-pinot-server-0 my-pinot-server-1 my-pinot-controller-1 controller server [Code Root Fixup: 0.0 ms]
[Other: 2.3 ms]
my-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
my-pinot-controller-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29827__20210121T1114Z
my-pinot-server-1 server my-pinot-server-0 server [Choose CSet: 0.2 ms]
my-pinot-server-0 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29828__20210121T1114Z
controller my-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29829__20210121T1115Z
my-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29830__20210121T1115Z
my-pinot-server-1 server [Ref Proc: 0.3 ms]
[Clear CT: 0.1 ms]
my-pinot-server-0 server my-pinot-controller-1 controller [Other: 0.5 ms]
my-pinot-controller-1 controller 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29831__20210121T1115Z
my-pinot-server-1 serverserver [Ref Enq: 0.0 ms]
my-pinot-server-0 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29832__20210121T1115Z
server my-pinot-server-1 server 2021/01/21 17:22:51.961 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29833__20210121T1115Z
my-pinot-server-1 server [Redirty Cards: 0.1 ms]
my-pinot-server-0 [Choose CSet: 0.0 ms]
2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29834__20210121T1115Z
my-pinot-server-1 server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29835__20210121T1115Z
my-pinot-server-1 my-pinot-controller-1 controller server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29836__20210121T1115Z
my-pinot-server-1 server [Humongous Register: 0.1 ms]
server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29837__20210121T1115Z
my-pinot-server-0 my-pinot-server-1 serverserver 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29838__20210121T1116Z
my-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29839__20210121T1116Z
my-pinot-server-1 [Humongous Reclaim: 0.1 ms]
server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29840__20210121T1116Z
my-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29841__20210121T1116Z
my-pinot-server-0 [Ref Proc: 0.1 ms]
server [Free CSet: 1.1 ms]
my-pinot-server-0 server [Eden: 200.0M(200.0M)->0.0B(195.0M) Survivors: 4096.0K->9216.0K Heap: 2471.9M(4096.0M)->1820.5M(4096.0M)]
my-pinot-server-0 server my-pinot-server-1 [Times: user=0.18 sys=0.01, real=0.03 secs]
server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29842__20210121T1116Z
my-pinot-server-1 server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29843__20210121T1116Z
my-pinot-controller-1 my-pinot-server-0 my-pinot-server-1server 2021-01-21T17:25:07.337+0000: 168.542: Total time for which application threads were stopped: 0.0329319 seconds, Stopping threads took: 0.0005259 seconds
my-pinot-server-0 server controller [Ref Enq: 0.0 ms]
my-pinot-controller-1server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29844__20210121T1116Z
my-pinot-server-1 server controller 2021-01-21T17:25:07.632+0000: 168.836: Application time: 0.2943829 seconds
my-pinot-server-0 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29845__20210121T1117Z
my-pinot-server-1 server server 2021-01-21T17:25:07.632+0000: 168.837: [GC pause (G1 Evacuation Pause) (mixed), 0.0321493 secs]
[Redirty Cards: 0.1 ms]
my-pinot-server-0 server [Parallel Time: 28.6 ms, GC Workers: 6]
my-pinot-server-0 server [GC Worker Start (ms): Min: 168838.0, Avg: 168838.0, Max: 168838.0, Diff: 0.1]
my-pinot-controller-1 controller [Humongous Register: 0.0 ms]
my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms]
my-pinot-server-0 my-pinot-controller-1 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.3, Max: 1.5, Diff: 0.2, Sum: 8.1]
my-pinot-server-0 server [Update RS (ms): Min: 9.7, Avg: 9.8, Max: 9.8, Diff: 0.1, Sum: 58.6]
my-pinot-server-0 controllerserver [Free CSet: 0.0 ms]
[Processed Buffers: Min: 67, Avg: 79.5, Max: 89, Diff: 22, Sum: 477]
2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29846__20210121T1117Z
my-pinot-server-1 my-pinot-server-0 my-pinot-controller-1 server [Scan RS (ms): Min: 1.8, Avg: 1.8, Max: 1.9, Diff: 0.1, Sum: 10.9]
server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29847__20210121T1117Z
controller [Eden: 5120.0K(91136.0K)->0.0B(152.0M) Survivors: 2048.0K->3072.0K Heap: 461.0M(1024.0M)->453.5M(1024.0M)]
my-pinot-server-0 my-pinot-controller-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Object Copy (ms): Min: 15.4, Avg: 15.5, Max: 15.6, Diff: 0.2, Sum: 92.9]
my-pinot-server-0 servermy-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29848__20210121T1117Z
my-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29849__20210121T1117Z
my-pinot-server-1 controllerserver [Times: user=0.04 sys=0.00, real=0.01 secs]
my-pinot-controller-1 controller2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29850__20210121T1117Z
2021-01-21T17:25:04.137+0000: 1350.530: [GC concurrent-root-region-scan-start]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 my-pinot-server-0 controller 2021-01-21T17:25:04.137+0000: 1350.530: Total time for which application threads were stopped: 0.0116309 seconds, Stopping threads took: 0.0001002 seconds
server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
my-pinot-server-0 my-pinot-server-1my-pinot-controller-1 controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-root-region-scan-end, 0.0031610 secs]
server 2021/01/21 17:22:51.966 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29851__20210121T1117Z
my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.966 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29852__20210121T1118Z
my-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29853__20210121T1118Z
server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-server-1controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-mark-start]
my-pinot-server-0 server [GC Worker Total (ms): Min: 28.4, Avg: 28.4, Max: 28.5, Diff: 0.1, Sum: 170.6]
my-pinot-server-0 server my-pinot-controller-1 controller server 2021-01-21T17:25:04.256+0000: 1350.649: Application time: 0.1189493 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.256+0000: 1350.649: [GC pause (G1 Evacuation Pause) (young), 0.0236359 secs]
my-pinot-controller-1 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29854__20210121T1118Z
controller [Parallel Time: 23.0 ms, GC Workers: 4]
my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350649.5, Avg: 1350649.5, Max: 1350649.5, Diff: 0.1]
[GC Worker End (ms): Min: 168866.4, Avg: 168866.4, Max: 168866.4, Diff: 0.0]
my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29855__20210121T1118Z
[Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.2, Diff: 0.2, Sum: 4.2]
my-pinot-controller-1 controller [Update RS (ms): Min: 6.5, Avg: 6.6, Max: 6.6, Diff: 0.1, Sum: 26.2]
my-pinot-controller-1 controller [Processed Buffers: Min: 65, Avg: 70.0, Max: 74, Diff: 9, Sum: 280]
my-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Object Copy (ms): Min: 15.2, Avg: 15.2, Max: 15.3, Diff: 0.2, Sum: 61.0]
my-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-controller-1 controllermy-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29856__20210121T1118Z
my-pinot-server-0 my-pinot-server-1server [Code Root Fixup: 0.0 ms]
my-pinot-server-0 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-controller-1 controller [GC Worker Total (ms): Min: 22.9, Avg: 22.9, Max: 22.9, Diff: 0.0, Sum: 91.6]
server server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29857__20210121T1118Z
my-pinot-server-1 [Code Root Purge: 0.0 ms]
my-pinot-server-0 server [Clear CT: 0.4 ms]
my-pinot-server-0 server [Other: 3.2 ms]
my-pinot-server-0 server [Choose CSet: 0.4 ms]
my-pinot-server-0 server [Ref Proc: 0.8 ms]
my-pinot-server-0 server [Ref Enq: 0.0 ms]
my-pinot-server-0 server [Redirty Cards: 0.1 ms]
my-pinot-server-0 server [Humongous Register: 0.1 ms]
my-pinot-server-0 server [Humongous Reclaim: 0.1 ms]
my-pinot-server-0 server [Free CSet: 1.2 ms]
my-pinot-server-0 server [Eden: 195.0M(195.0M)->0.0B(192.0M) Survivors: 9216.0K->12288.0K Heap: 2064.8M(4096.0M)->1443.5M(4096.0M)]
my-pinot-server-0 server [Times: user=0.17 sys=0.00, real=0.03 secs]
my-pinot-server-0 server 2021-01-21T17:25:07.665+0000: 168.870: Total time for which application threads were stopped: 0.0330694 seconds, Stopping threads took: 0.0002922 seconds
my-pinot-server-0 server 2021-01-21T17:25:07.829+0000: 169.034: Application time: 0.1644559 seconds
my-pinot-server-0 server 2021-01-21T17:25:07.838+0000: 169.043: [GC pause (G1 Evacuation Pause) (mixed), 0.0260221 secs]
my-pinot-server-0 server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29858__20210121T1118Z
my-pinot-server-1my-pinot-controller-1 server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29859__20210121T1119Z
my-pinot-server-1 controller [GC Worker End (ms): Min: 1350672.4, Avg: 1350672.4, Max: 1350672.4, Diff: 0.0]
my-pinot-controller-1 controller [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controller server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29860__20210121T1119Z
my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29861__20210121T1119Z
[Code Root Purge: 0.0 ms]
my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29862__20210121T1119Z
my-pinot-server-1 my-pinot-controller-1 controller [Clear CT: 0.1 ms]
my-pinot-controller-1server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29863__20210121T1119Z
my-pinot-server-1 server server [Parallel Time: 23.6 ms, GC Workers: 6]
my-pinot-server-0 server [GC Worker Start (ms): Min: 169043.2, Avg: 169043.2, Max: 169043.3, Diff: 0.1]
my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.5, Max: 1.6, Diff: 0.2, Sum: 8.9]
my-pinot-server-0 server [Update RS (ms): Min: 13.8, Avg: 13.8, Max: 13.8, Diff: 0.1, Sum: 82.8]
my-pinot-server-0 server [Processed Buffers: Min: 50, Avg: 54.3, Max: 58, Diff: 8, Sum: 326]
my-pinot-server-0 server [Scan RS (ms): Min: 0.2, Avg: 0.3, Max: 0.3, Diff: 0.1, Sum: 1.6]
2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29864__20210121T1119Z
my-pinot-server-1my-pinot-server-0 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29865__20210121T1119Z
my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29866__20210121T1120Z
my-pinot-server-1 server controller2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29867__20210121T1120Z
my-pinot-server-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Object Copy (ms): Min: 7.7, Avg: 7.9, Max: 8.0, Diff: 0.2, Sum: 47.2]
my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.2]
my-pinot-server-0 server [GC Worker Total (ms): Min: 23.4, Avg: 23.4, Max: 23.5, Diff: 0.1, Sum: 140.7]
[Other: 0.6 ms]
my-pinot-controller-1 server my-pinot-server-0 server controller [Choose CSet: 0.0 ms]
my-pinot-controller-1 controller [Ref Proc: 0.1 ms]
[GC Worker End (ms): Min: 169066.7, Avg: 169066.7, Max: 169066.7, Diff: 0.0]
my-pinot-server-0 2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29868__20210121T1120Z
my-pinot-server-1 server 2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29869__20210121T1120Z
server my-pinot-server-1 [Code Root Fixup: 0.0 ms]
my-pinot-server-0 server [Code Root Purge: 0.0 ms]
my-pinot-server-0 server server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29870__20210121T1120Z
my-pinot-server-1 my-pinot-controller-1 controller [Ref Enq: 0.0 ms]
[Clear CT: 0.2 ms]
my-pinot-controller-1 controller my-pinot-server-0 server [Redirty Cards: 0.1 ms]
my-pinot-controller-1 [Other: 2.2 ms]
my-pinot-server-0 controller server [Choose CSet: 0.0 ms]
[Humongous Register: 0.0 ms]
my-pinot-server-0 server [Ref Proc: 1.0 ms]
my-pinot-server-0 server [Ref Enq: 0.0 ms]
my-pinot-server-0 server [Redirty Cards: 0.1 ms]
my-pinot-server-0 my-pinot-controller-1 controller server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29871__20210121T1120Z
my-pinot-server-1 server [Humongous Register: 0.2 ms]
my-pinot-server-0 server [Humongous Reclaim: 0.1 ms]
my-pinot-server-0 server [Free CSet: 0.3 ms]
server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29872__20210121T1120Z
my-pinot-server-1 server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29873__20210121T1120Z
my-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29874__20210121T1120Z
my-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29875__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.974 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29876__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29877__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29878__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29879__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29880__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29881__20210121T1121Z
my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29882__20210121T1121Z
my-pinot-server-1 [Humongous Reclaim: 0.1 ms]
my-pinot-controller-1 controller [Free CSet: 0.1 ms]
my-pinot-controller-1 controller [Eden: 152.0M(152.0M)->0.0B(298.0M) Survivors: 3072.0K->20480.0K Heap: 638.1M(1024.0M)->482.3M(1024.0M)]
my-pinot-controller-1 controller [Times: user=0.09 sys=0.00, real=0.03 secs]
my-pinot-controller-1 controller 2021-01-21T17:25:04.280+0000: 1350.673: Total time for which application threads were stopped: 0.0241836 seconds, Stopping threads took: 0.0000443 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.955: [GC concurrent-mark-end, 0.4221306 secs]
my-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29883__20210121T1122Z
my-pinot-server-0 my-pinot-server-1server [Eden: 192.0M(192.0M)->0.0B(2155.0M) Survivors: 12288.0K->23552.0K Heap: 1650.1M(4096.0M)->1449.3M(4096.0M)]
my-pinot-server-0 controllerserver [Times: user=0.14 sys=0.00, real=0.03 secs]
my-pinot-server-0 server 2021-01-21T17:25:07.864+0000: 169.069: Total time for which application threads were stopped: 0.0349989 seconds, Stopping threads took: 0.0082034 seconds
my-pinot-server-0 server 2021-01-21T17:25:09.808+0000: 171.013: Application time: 1.9435757 seconds
my-pinot-server-0 serverserver 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29884__20210121T1122Z
my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29885__20210121T1122Z
2021-01-21T17:25:09.813+0000: 171.018: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0428457 secs]
my-pinot-server-0 2021-01-21T17:25:04.562+0000: 1350.955: Application time: 0.2822203 seconds
server [Parallel Time: 37.6 ms, GC Workers: 6]
my-pinot-server-0 my-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.956: [GC remark 2021-01-21T17:25:04.562+0000: 1350.956: [Finalize Marking, 0.0002598 secs] 2021-01-21T17:25:04.563+0000: 1350.956: [GC ref-proc, 0.0004448 secs] 2021-01-21T17:25:04.563+0000: 1350.957: [Unloading, 0.0095395 secs], 0.0108947 secs]
my-pinot-controller-1 my-pinot-server-1 controller [Times: user=0.04 sys=0.00, real=0.01 secs]
my-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Total time for which application threads were stopped: 0.0114314 seconds, Stopping threads took: 0.0000926 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Application time: 0.0000628 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.574+0000: 1350.967: [GC cleanup 774M->655M(1024M), 0.0018691 secs]
my-pinot-controller-1 controller [Times: user=0.01 sys=0.00, real=0.00 secs]
my-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29886__20210121T1122Z
controller my-pinot-server-1 server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29887__20210121T1122Z
server [GC Worker Start (ms): Min: 171018.3, Avg: 171018.4, Max: 171018.4, Diff: 0.1]
my-pinot-server-0 my-pinot-server-1server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.4, Max: 1.6, Diff: 0.2, Sum: 8.6]
my-pinot-server-0 server [Update RS (ms): Min: 20.6, Avg: 20.7, Max: 20.8, Diff: 0.1, Sum: 124.3]
my-pinot-server-0 server [Processed Buffers: Min: 144, Avg: 149.8, Max: 159, Diff: 15, Sum: 899]
my-pinot-server-0 2021-01-21T17:25:04.576+0000: 1350.969: Total time for which application threads were stopped: 0.0023571 seconds, Stopping threads took: 0.0000678 seconds
my-pinot-controller-1 server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29888__20210121T1122Z
my-pinot-server-1 controllerserver [Scan RS (ms): Min: 0.4, Avg: 0.4, Max: 0.5, Diff: 0.1, Sum: 2.6]
server 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-start]
my-pinot-controller-1 my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-server-0 server [Object Copy (ms): Min: 14.5, Avg: 14.7, Max: 14.8, Diff: 0.2, Sum: 88.2]
my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6]
my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
my-pinot-server-0 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29889__20210121T1122Z
my-pinot-server-1 controller 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-end, 0.0001739 secs]
server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29890__20210121T1123Z
my-pinot-controller-1 server [GC Worker Total (ms): Min: 37.3, Avg: 37.4, Max: 37.4, Diff: 0.1, Sum: 224.2]
my-pinot-server-0 server [GC Worker End (ms): Min: 171055.7, Avg: 171055.7, Max: 171055.8, Diff: 0.1]
my-pinot-server-0 server [Code Root Fixup: 0.0 ms]
my-pinot-server-0 server controller 2021-01-21T17:25:04.654+0000: 1351.048: Application time: 0.0785958 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.655+0000: 1351.048: [GC pause (G1 Evacuation Pause) (young), 0.0666046 secs]
my-pinot-server-1 [Code Root Purge: 0.0 ms]
server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29891__20210121T1123Z
my-pinot-server-1 my-pinot-controller-1 controller [Parallel Time: 65.5 ms, GC Workers: 4]
my-pinot-controller-1 my-pinot-server-0 server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29892__20210121T1123Z
server [Clear CT: 0.5 ms]
my-pinot-server-0 server [Other: 4.8 ms]
my-pinot-server-0 server [Choose CSet: 0.0 ms]
my-pinot-server-0 server [Ref Proc: 2.2 ms]
my-pinot-server-0 server [Ref Enq: 0.0 ms]
my-pinot-server-0 server [Redirty Cards: 0.1 ms]
my-pinot-server-0 server [Humongous Register: 0.1 ms]
my-pinot-server-0 server [Humongous Reclaim: 0.6 ms]
controller my-pinot-server-0 [GC Worker Start (ms): Min: 1351048.4, Avg: 1351048.4, Max: 1351048.4, Diff: 0.0]
server [Free CSet: 1.0 ms]
my-pinot-controller-1 my-pinot-server-1 my-pinot-server-0 server [Eden: 1386.0M(2155.0M)->0.0B(2134.0M) Survivors: 23552.0K->48128.0K Heap: 3188.4M(4096.0M)->1474.4M(4096.0M)]
my-pinot-server-0 server [Times: user=0.23 sys=0.00, real=0.04 secs]
my-pinot-server-0 server controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.6]
my-pinot-controller-1 controller server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29893__20210121T1123Z
2021/01/21 17:25:10.641 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=mylog__0__13120__20210119T1725Z, expected=ONLINE, found=OFFLINE, creationTime=1611249776061, modifiedTime=1611249910406, version=1030, waitingFor=CurrentStateMatch, resource=mylog_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
my-pinot-server-0 [Update RS (ms): Min: 16.0, Avg: 16.0, Max: 16.0, Diff: 0.1, Sum: 63.9]
server 2021-01-21T17:25:09.856+0000: 2021-01-21T17:25:09.856+0000: 171.061: 171.061: [GC concurrent-root-region-scan-start]
my-pinot-server-0 server my-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 111.0, Max: 117, Diff: 14, Sum: 444]
my-pinot-server-1 server Total time for which application threads were stopped: 0.0480709 seconds, Stopping threads took: 0.0044806 seconds
my-pinot-server-0 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-root-region-scan-end, 0.0198202 secs]
2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29894__20210121T1123Z
my-pinot-server-1 my-pinot-controller-1 controller [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.1, Diff: 0.0, Sum: 0.3]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1server 2021/01/21 17:22:51.980 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29895__20210121T1123Z
my-pinot-server-1 server 2021/01/21 17:22:51.980 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29896__20210121T1123Z
my-pinot-server-1 controller [Object Copy (ms): Min: 48.1, Avg: 48.2, Max: 48.3, Diff: 0.2, Sum: 192.8]
server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29897__20210121T1124Z
my-pinot-server-1 my-pinot-server-0servermy-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-controller-1 controller [GC Worker Total (ms): Min: 65.4, Avg: 65.4, Max: 65.4, Diff: 0.0, Sum: 261.7]
my-pinot-controller-1 controller [GC Worker End (ms): Min: 1351113.8, Avg: 1351113.8, Max: 1351113.8, Diff: 0.0]
my-pinot-controller-1 controller [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controller [Code Root Purge: 0.0 ms]
my-pinot-controller-1 controller [Clear CT: 0.1 ms]
my-pinot-controller-1 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-mark-start]
my-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: [GC concurrent-mark-end, 1.3475822 secs]
2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29898__20210121T1124Z
my-pinot-server-1 my-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: Application time: 1.3674839 seconds
server my-pinot-server-0 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29899__20210121T1124Z
my-pinot-server-1 controller [Other: 1.0 ms]
my-pinot-controller-1 controllerserver 2021-01-21T17:25:11.228+0000: 172.433: [GC remark 2021-01-21T17:25:11.228+0000: 172.433: [Finalize Marking, 0.0003440 secs] 2021-01-21T17:25:11.229+0000: 172.434: [GC ref-proc, 0.0850389 secs] 2021-01-21T17:25:11.314+0000: 172.519: [Unloading, 0.0079844 secs], 0.0957425 secs]
my-pinot-server-0 server [Times: user=0.11 sys=0.03, real=0.10 secs]
my-pinot-server-0 server 2021-01-21T17:25:11.324+0000: 172.529: Total time for which application threads were stopped: 0.1009431 seconds, Stopping threads took: 0.0045995 seconds
server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29900__20210121T1124Z
[Choose CSet: 0.0 ms]
my-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29901__20210121T1124Z
my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29902__20210121T1124Z
my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29903__20210121T1125Z
my-pinot-server-1 controller2021-01-21T17:25:11.324+0000: 172.529: Application time: 0.0000758 seconds
my-pinot-server-0 server server 2021-01-21T17:25:11.325+0000: 172.530: [GC cleanup 2780M->2780M(4096M), 0.0035921 secs]
my-pinot-server-0 [Ref Proc: 0.2 ms]
my-pinot-controller-1 controller server [Ref Enq: 0.0 ms]
my-pinot-controller-1 controller [Redirty Cards: 0.3 ms]
[Times: user=0.02 sys=0.00, real=0.00 secs]
my-pinot-controller-1 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29904__20210121T1125Z
my-pinot-server-1 servermy-pinot-server-0 server 2021-01-21T17:25:11.328+0000: 172.533: Total time for which application threads were stopped: 0.0041608 seconds, Stopping threads took: 0.0000167 seconds
my-pinot-server-0 server 2021/01/21 17:22:51.982 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29905__20210121T1125Z
my-pinot-server-1 2021-01-21T17:25:11.481+0000: 172.686: Application time: 0.1529441 seconds
servercontrollermy-pinot-server-0 server 2021-01-21T17:25:11.482+0000: 172.687: Total time for which application threads were stopped: 0.0005239 seconds, Stopping threads took: 0.0000613 seconds
my-pinot-server-0 [Humongous Register: 0.0 ms]
servermy-pinot-controller-1 2021/01/21 17:22:51.982 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29906__20210121T1125Z
my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29907__20210121T1125Z
my-pinot-server-1 controller server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29908__20210121T1125Z
my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29909__20210121T1125Z
[Humongous Reclaim: 0.1 ms]
my-pinot-controller-1my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29910__20210121T1125Z
my-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29911__20210121T1126Z
controller [Free CSet: 0.2 ms]
my-pinot-controller-1 controller [Eden: 298.0M(298.0M)->0.0B(11264.0K) Survivors: 20480.0K->40960.0K Heap: 700.6M(1024.0M)->429.5M(1024.0M)]
my-pinot-controller-1 controller [Times: user=0.26 sys=0.00, real=0.07 secs]
my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29912__20210121T1126Z
my-pinot-server-1 server 2021-01-21T17:25:11.489+0000: 172.694: Application time: 0.0071873 seconds
my-pinot-server-0 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29913__20210121T1126Z
my-pinot-server-1 server 2021-01-21T17:25:11.490+0000: 172.694: Total time for which application threads were stopped: 0.0004430 seconds, Stopping threads took: 0.0000272 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.551+0000: 172.755: Application time: 0.0610437 seconds
2021-01-21T17:25:04.722+0000: 1351.115: Total time for which application threads were stopped: 0.0671829 seconds, Stopping threads took: 0.0000684 seconds
my-pinot-controller-1 my-pinot-server-0controller 2021-01-21T17:25:04.737+0000: 1351.130: Application time: 0.0150208 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.737+0000: 1351.130: [GC pause (G1 Evacuation Pause) (mixed), 0.0775762 secs]
server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29914__20210121T1126Z
server 2021-01-21T17:25:11.552+0000: 172.757: Total time for which application threads were stopped: 0.0013581 seconds, Stopping threads took: 0.0000241 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.710+0000: 172.915: Application time: 0.1581764 seconds
my-pinot-server-0 my-pinot-server-1 server my-pinot-controller-1 controller [Parallel Time: 76.1 ms, GC Workers: 4]
my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1351130.8, Avg: 1351130.8, Max: 1351130.8, Diff: 0.1]
my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.2]
my-pinot-controller-1 controller [Update RS (ms): Min: 31.3, Avg: 31.4, Max: 31.4, Diff: 0.1, Sum: 125.4]
my-pinot-controller-1 controller [Processed Buffers: Min: 124, Avg: 125.0, Max: 126, Diff: 2, Sum: 500]
my-pinot-controller-1 server controller 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29915__20210121T1126Z
my-pinot-server-1 2021-01-21T17:25:11.718+0000: 172.923: Total time for which application threads were stopped: 0.0080070 seconds, Stopping threads took: 0.0074375 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.724+0000: 172.928: Application time: 0.0054288 seconds
my-pinot-server-0server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29916__20210121T1126Z
my-pinot-server-1 server 2021/01/21 17:22:51.986 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29917__20210121T1126Z
my-pinot-server-1 [Scan RS (ms): Min: 13.4, Avg: 13.5, Max: 13.7, Diff: 0.3, Sum: 54.1]
my-pinot-controller-1 controllerserver [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5]
2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29918__20210121T1127Z
my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29919__20210121T1127Z
controller [Object Copy (ms): Min: 29.8, Avg: 29.9, Max: 30.1, Diff: 0.3, Sum: 119.8]
my-pinot-controller-1 server controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29920__20210121T1127Z
my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29921__20210121T1127Z
2021-01-21T17:25:11.728+0000: 172.933: Total time for which application threads were stopped: 0.0044722 seconds, Stopping threads took: 0.0031338 seconds
my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29922__20210121T1127Z
my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29923__20210121T1127Z
my-pinot-controller-1 my-pinot-server-0 server controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
my-pinot-controller-1 2021-01-21T17:25:11.732+0000: 172.937: Application time: 0.0044179 seconds
my-pinot-server-0 controllerserver my-pinot-server-1 [GC Worker Total (ms): Min: 76.0, Avg: 76.0, Max: 76.0, Diff: 0.0, Sum: 304.1]
my-pinot-controller-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29924__20210121T1127Z
my-pinot-server-1 controller [GC Worker End (ms): Min: 1351206.8, Avg: 1351206.8, Max: 1351206.8, Diff: 0.0]
server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29925__20210121T1127Z
my-pinot-server-1 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0004491 seconds, Stopping threads took: 0.0000215 seconds
my-pinot-server-0 server my-pinot-controller-1 controller 2021-01-21T17:25:11.733+0000: 172.938: Application time: 0.0000363 seconds
my-pinot-server-0 [Code Root Fixup: 0.0 ms]
my-pinot-controller-1 controllerserver 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0003160 seconds, Stopping threads took: 0.0000079 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Application time: 0.0124480 seconds
[Code Root Purge: 0.0 ms]
my-pinot-controller-1 server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29926__20210121T1128Z
my-pinot-server-1 controller [Clear CT: 0.1 ms]
server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29927__20210121T1128Z
my-pinot-server-1 server my-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Total time for which application threads were stopped: 0.0003742 seconds, Stopping threads took: 0.0000195 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.759+0000: 172.964: Application time: 0.0131737 seconds
my-pinot-server-02021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29928__20210121T1128Z
my-pinot-server-1 server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29929__20210121T1128Z
my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29930__20210121T1128Z
my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29931__20210121T1128Z
my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29932__20210121T1128Z
my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29933__20210121T1129Z
my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29934__20210121T1129Z
server my-pinot-server-1 controller [Other: 1.3 ms]
my-pinot-controller-1 controller [Choose CSet: 0.2 ms]
my-pinot-controller-1 controller [Ref Proc: 0.1 ms]
my-pinot-controller-1 controller [Ref Enq: 0.0 ms]
my-pinot-controller-1 2021-01-21T17:25:11.760+0000: 172.965: Total time for which application threads were stopped: 0.0007156 seconds, Stopping threads took: 0.0002863 seconds
controller [Redirty Cards: 0.4 ms]
my-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Application time: 0.0087255 seconds
my-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Total time for which application threads were stopped: 0.0007774 seconds, Stopping threads took: 0.0002464 seconds
my-pinot-controller-1 controller [Humongous Register: 0.0 ms]
my-pinot-controller-1 my-pinot-server-0 server 2021-01-21T17:25:11.771+0000: 172.976: Application time: 0.0015455 seconds
controller my-pinot-server-0 server 2021-01-21T17:25:11.773+0000: 172.977: Total time for which application threads were stopped: 0.0015423 seconds, Stopping threads took: 0.0002360 seconds
my-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: Application time: 0.5611696 seconds
my-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: [GC pause (G1 Evacuation Pause) (young), 0.2155960 secs]
my-pinot-server-0 [Humongous Reclaim: 0.0 ms]
servermy-pinot-controller-1 controller [Free CSet: 0.4 ms]
my-pinot-controller-1 controller [Eden: 11264.0K(11264.0K)->0.0B(48128.0K) Survivors: 40960.0K->4096.0K Heap: 455.0M(1024.0M)->354.2M(1024.0M)]
my-pinot-controller-1 [Parallel Time: 208.2 ms, GC Workers: 6]
my-pinot-server-0 server server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29935__20210121T1129Z
[GC Worker Start (ms): Min: 173539.5, Avg: 173539.5, Max: 173539.5, Diff: 0.1]
my-pinot-server-1 controller server my-pinot-server-0 [Times: user=0.30 sys=0.00, real=0.07 secs]
server [Ext Root Scanning (ms): Min: 1.1, Avg: 1.3, Max: 1.6, Diff: 0.4, Sum: 7.7]
2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29936__20210121T1129Z
my-pinot-server-1 my-pinot-controller-1 controller 2021-01-21T17:25:04.815+0000: 1351.208: Total time for which application threads were stopped: 0.0781603 seconds, Stopping threads took: 0.0000425 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.841+0000: 1351.235: Application time: 0.0263536 seconds
my-pinot-controller-1 controller 2021-01-21T17:25:04.842+0000: 1351.235: [GC pause (G1 Evacuation Pause) (mixed), 0.0820991 secs]
server my-pinot-controller-1 controller 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29937__20210121T1129Z
my-pinot-server-1 [Parallel Time: 81.1 ms, GC Workers: 4]
my-pinot-server-0 my-pinot-controller-1 server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29938__20210121T1129Z
controller [GC Worker Start (ms): Min: 1351235.2, Avg: 1351235.2, Max: 1351235.2, Diff: 0.0]
my-pinot-controller-1 servercontroller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.3]
my-pinot-controller-1 [Update RS (ms): Min: 15.5, Avg: 15.5, Max: 15.6, Diff: 0.1, Sum: 93.3]
my-pinot-server-0 server [Processed Buffers: Min: 108, Avg: 112.2, Max: 115, Diff: 7, Sum: 673]
my-pinot-server-0 server [Scan RS (ms): Min: 166.2, Avg: 166.3, Max: 166.5, Diff: 0.2, Sum: 998.0]
my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29939__20210121T1129Z
my-pinot-server-1 controller server [Update RS (ms): Min: 35.6, Avg: 35.7, Max: 35.9, Diff: 0.4, Sum: 142.9]
my-pinot-controller-1 controller [Processed Buffers: Min: 150, Avg: 155.0, Max: 159, Diff: 9, Sum: 620]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
my-pinot-controller-1 controller [Scan RS (ms): Min: 5.9, Avg: 14.1, Max: 22.5, Diff: 16.6, Sum: 56.4]
my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 4.2, Max: 16.4, Diff: 16.4, Sum: 16.8]
server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29940__20210121T1129Z
my-pinot-server-1 server 2021/01/21 17:22:51.991 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29941__20210121T1129Z
my-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29942__20210121T1129Z
my-pinot-controller-1 my-pinot-server-1 controller my-pinot-server-0 [Object Copy (ms): Min: 21.7, Avg: 25.9, Max: 38.2, Diff: 16.6, Sum: 103.5]
server server my-pinot-controller-1 controller 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29943__20210121T1130Z
my-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29944__20210121T1130Z
my-pinot-server-1 [Object Copy (ms): Min: 24.4, Avg: 24.8, Max: 25.0, Diff: 0.6, Sum: 148.7]
my-pinot-server-0 server server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
my-pinot-server-02021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29945__20210121T1130Z
my-pinot-server-1 server [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 7]
my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server [GC Worker Total (ms): Min: 208.0, Avg: 208.0, Max: 208.1, Diff: 0.1, Sum: 1248.2]
2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29946__20210121T1130Z
my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29947__20210121T1130Z
my-pinot-server-1 my-pinot-server-0 server [GC Worker End (ms): Min: 173747.5, Avg: 173747.5, Max: 173747.6, Diff: 0.1]
server my-pinot-server-0 server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29948__20210121T1130Z
my-pinot-server-1 controller [Termination Attempts: Min: 1, Avg: 5.8, Max: 10, Diff: 9, Sum: 23]
my-pinot-controller-1 [Code Root Fixup: 0.0 ms]
my-pinot-server-0 server controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29949__20210121T1130Z
my-pinot-server-1 server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29950__20210121T1130Z
my-pinot-server-1 server 2021/01/21 17:22:51.995 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29951__20210121T1130Z
my-pinot-server-1 server my-pinot-controller-1 controller [GC Worker Total (ms): Min: 81.0, Avg: 81.0, Max: 81.0, Diff: 0.0, Sum: 324.0]
[Code Root Purge: 0.0 ms]
2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29952__20210121T1130Z
my-pinot-server-1 my-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29953__20210121T1130Z
my-pinot-controller-1 controller [GC Worker End (ms): Min: 1351316.2, Avg: 1351316.2, Max: 1351316.2, Diff: 0.0]
my-pinot-controller-1 [Clear CT: 0.9 ms]
controller my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29954__20210121T1130Z
my-pinot-server-0 my-pinot-server-1 server [Other: 6.5 ms]
[Code Root Fixup: 0.0 ms]
my-pinot-controller-1 my-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29955__20210121T1130Z
[Choose CSet: 0.0 ms]
my-pinot-server-1 server my-pinot-server-0 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29956__20210121T1131Z
server [Ref Proc: 3.8 ms]
my-pinot-server-0 server [Ref Enq: 0.0 ms]
my-pinot-server-0 server [Redirty Cards: 0.2 ms]
my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29957__20210121T1131Z
my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29958__20210121T1131Z
my-pinot-server-1 controller [Code Root Purge: 0.0 ms]
my-pinot-controller-1 controller [Clear CT: 0.1 ms]
server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29959__20210121T1131Z
server my-pinot-server-1 server [Humongous Register: 0.2 ms]
my-pinot-server-0 my-pinot-controller-1 controller 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29960__20210121T1131Z
my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29961__20210121T1131Z
my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29962__20210121T1132Z
my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29963__20210121T1132Z
my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29964__20210121T1132Z
[Other: 0.8 ms]
server [Humongous Reclaim: 0.7 ms]
my-pinot-server-0 my-pinot-controller-1 server [Free CSet: 1.4 ms]
controller my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29965__20210121T1132Z
[Choose CSet: 0.1 ms]
my-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29966__20210121T1132Z
my-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29967__20210121T1132Z
my-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29968__20210121T1132Z
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765591119
@subourbonite I am running two controllers and two brokers not aware that the controller data should be shared . I can map a same PVC for both controllers. Thanks for pointing it out.
@fx19880617
a) it is good to know that I can use S3 for controller storage and use it as a backup. Could you share some documents related to setting it up. is it same as the deep store functionality?
b) Hope there is a PR going on related to minion task?
c) I tried restarting servers in round robin fashion to avoid outage and that didn't help. I had to restart the entire cluster at same time to make it work again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-767933847
@fx19880617 , Other than adding configs to controller and server I can see a Job spec as well. Why we need a Job spec? I was expecting controller to push segment directly to s3 and download directly from s3 whenever required without any job. Is it possible to implement S3 backend without Job?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-769332486
Thanks @fx19880617 , will try the CRON minion and see. Also as @subourbonite mentioned I mapped the stateful set for controller to use single pvc mount. So controller data is shared across all controller pods. Also I mapped the controller to use an EFS with no restriction in data size. Now my question now is
As the controller pvc have a copy of all segments will it suffice as a backup if one of the Pinot Server SSD fails? If one of the Pinot Server SSD fails and if I replace the disk will the server catches up the lost segments from controller EFS automatically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-762385437
@fx19880617 , Max open file limit was unlimited however I had to update vm max map count in os. After that I restarted Pinot cluster and this came right after the restart of Pinot Server. But it disappeared after 10 mts and cluster recovered automatically. It is good to know that there is a periodical check to fix the segments. Most of these segments in the ERROR list are completed ones. Do you think I can consider the Cluster healthy then?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765873995
@subourbonite At the moment the helm chart in the repo uses Statefulset for controller. which always creates separate storage. The folder in the chart for controller is /var/pinot/controller/data. And I used the chart to create the Cluster.
Also how can I set controller data to S3, I am assuming that involves a whole set of different configuration?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-768186395
> @fx19880617 , Other than adding configs to controller and server I can see a Job spec as well. Is Job spec mandatory? I was expecting controller to push segment directly to s3 and download directly from s3 whenever required without any job. Is it possible to implement S3 backend without Job?
If you deploy Pinot minion and enable the CRON scheduler and set the cron expression in table config, then it's possible :)
Ref PR: https://github.com/apache/incubator-pinot/pull/6451
I'm also adding the corresponding documentation for this feature into pinot docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765110529
@fx19880617 do you mean the controller data folder? I have mapped /var/pinot/controller/data to a SSD and its max size is 10GB however it was only 50% full. I am using 2 controller and both were just around 50% full.
I have some questions:
a) Is it ok if I move this controller data folder to EFS? So it will grow automatically. Or will it cause any latency?. Please note for pinot server I will continue using SSD for performance. I am planning to use EFS just for controller data.
b) I have to create small segments because the textindex which I am relying heavily. Textindex have an issue where it only scans online or completed segments(https://github.com/apache/incubator-pinot/issues/6431) . Because of this behavior I am creating small segments so that I can get instant results. Is there any way I can merge completed segments with some periodic process?
c) Once I restarted controller, broker and server at the same time the issue went away and ingestion started working. However individual restart did not helped. Why this is so?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-768113120
@fx19880617 , I followed the steps and tried to modify JAVA_OPTS to add the se plugin. But it does break kafka. I am seeing the following error.
[main] {"code":500,"error":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory"}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-767304859
you can follow this doc to add configs for s3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-761973043
I think this is due to the max open file limit. Once you uplift it, this issue should just be resolved.
How long have you seen the real-time segments are in bad status? Pinot internally has a periodical check to fix the consuming segments errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6449: Errors after Pinot server ran out of max map count
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6449:
URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765591119
@subourbonite I am running two controllers and two brokers not aware that the controller data should be shared . I can map same PVC for both controllers. Thanks for pointing it out.
@fx19880617
a) it is good to know that I can use S3 for controller storage and use it as a backup. Could you share some documents related to setting it up. is it same as the deep store functionality?
b) Hope there is a PR going on related to minion task?
c) I tried restarting servers in round robin fashion to avoid outage and that didn't help. I had to restart the entire cluster at same time to make it work again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org