You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/10/25 02:48:26 UTC
[GitHub] [pinot] cxing opened a new issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
cxing opened a new issue #7626:
URL: https://github.com/apache/pinot/issues/7626
Suggest change:
Failed log:
```
2021-10-22 04:04:34.307 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread] Exception while executing a message. java.lang.RuntimeException: Caught exception while reloading segment: in table: wme_metrics_customer_report_REALTIME msgId: e881ac8e-f1e3-4533-85c5-4cd60565cc16 type: USER_DEFINE_MSG
java.lang.RuntimeException: Caught exception while reloading segment: in table: wme_metrics_customer_report_REALTIME
at org.apache.pinot.server.starter.helix.SegmentMessageHandlerFactory$SegmentReloadMessageHandler.handleMessage(SegmentMessageHandlerFactory.java:161) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: java.lang.IllegalStateException: Index directory: /vdb/pinot/data/wme_metrics_customer_report_REALTIME/wme_metrics_customer_report__24__333__20211001T0003Z is not a directory
at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.reloadSegment(HelixInstanceDataManager.java:262) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.reloadAllSegments(HelixInstanceDataManager.java:224) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.SegmentMessageHandlerFactory$SegmentReloadMessageHandler.handleMessage(SegmentMessageHandlerFactory.java:150) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
... 6 more
2021-10-22 04:04:34.366 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread] Message execution failed. msgId: e881ac8e-f1e3-4533-85c5-4cd60565cc16, errorMsg: Caught exception while reloading segment: in table: wme_metrics_customer_report_REALTIME
2021-10-22 04:04:34.375 ERROR [wme_metrics_customer_report_REALTIME-SegmentReloadMessageHandler] [HelixTaskExecutor-message_handle_thread] onError: INTERNAL, ERROR
java.lang.RuntimeException: Caught exception while reloading segment: in table: wme_metrics_customer_report_REALTIME
at org.apache.pinot.server.starter.helix.SegmentMessageHandlerFactory$SegmentReloadMessageHandler.handleMessage(SegmentMessageHandlerFactory.java:161) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: java.lang.IllegalStateException: Index directory: /vdb/pinot/data/wme_metrics_customer_report_REALTIME/wme_metrics_customer_report__24__333__20211001T0003Z is not a directory
at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.reloadSegment(HelixInstanceDataManager.java:262) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.reloadAllSegments(HelixInstanceDataManager.java:224) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
at org.apache.pinot.server.starter.helix.SegmentMessageHandlerFactory$SegmentReloadMessageHandler.handleMessage(SegmentMessageHandlerFactory.java:150) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-031903bac9a6b6fa3b906d6093925d5fdc2e5502]
... 6 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7626:
URL: https://github.com/apache/pinot/issues/7626#issuecomment-957017040
@andruszd When reloading fails, pinot will try to recover the segment to the previous state so that it doesn't cause data loss. When loading the segment for the first time, if it cannot be loaded, pinot will try to download a new copy from the deep store. If the segment is corrupted in the deep store (should be very rare, and easy to detect when segment is pushed), currently the user needs to manually delete it through the controller rest api, and probably regenerate the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #7626:
URL: https://github.com/apache/pinot/issues/7626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] andruszd commented on issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
andruszd commented on issue #7626:
URL: https://github.com/apache/pinot/issues/7626#issuecomment-952919302
How about also moving the failed segment out of the way into a holding directory so that it can be fixed or deleted in a safe way. Plus the app need to have a function to remove bad segments cleanly so that they don't show up as unhealthy in the metrics ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #7626:
URL: https://github.com/apache/pinot/issues/7626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7626:
URL: https://github.com/apache/pinot/issues/7626#issuecomment-957017040
@andruszd When reloading fails, pinot will try to recover the segment to the previous state so that it doesn't cause data loss. When loading the segment for the first time, if it cannot be loaded, pinot will try to download a new copy from the deep store. If the segment is corrupted in the deep store (should be very rare, and easy to detect when segment is pushed), currently the user needs to manually delete it through the controller rest api, and probably regenerate the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7626:
URL: https://github.com/apache/pinot/issues/7626#issuecomment-957017040
@andruszd When reloading fails, pinot will try to recover the segment to the previous state so that it doesn't cause data loss. When loading the segment for the first time, if it cannot be loaded, pinot will try to download a new copy from the deep store. If the segment is corrupted in the deep store (should be very rare, and easy to detect when segment is pushed), currently the user needs to manually delete it through the controller rest api, and probably regenerate the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #7626: Pinot need skip the failed segments to continue reload the rest of the segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #7626:
URL: https://github.com/apache/pinot/issues/7626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org