You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/12 20:44:05 UTC
[GitHub] [druid] pchang388 opened a new issue, #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
pchang388 opened a new issue, #12898:
URL: https://github.com/apache/druid/issues/12898
**Apache Druid Version:** 0.23.0
**Problem**:
Sometimes when we have infrastructure issues, we have to decommission some Historicals and temporarily lower retention period (to not run out of disk on running ones). When we bring back the Historicals and re-apply our old retention period and reload the dropped data, our ingestion tasks wait for handoff much longer and sometimes fail because they did not handoff completely in the completionTimeout phase even if that is set to a higher number.
Is it possible to adjust Coordinator behavior to prioritize new segment handoffs from currently running ingestion tasks over reloading dropped data so that running ingestion tasks are not affected and still have some segments being reloaded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1216388045
this seems like a genuine problem. @kfaraz is working on a proposal to improve the segment balancing - https://github.com/apache/druid/issues/12881. maybe this issue can be taken up too as part of that proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] danprince1 commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
danprince1 commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1315254296
We've had this running in a big cluster in production for a long time and it's been solid. I couldn't get traction on it when I submitted it; maybe I'll try again. https://github.com/apache/druid/pull/12013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
kfaraz commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1315457293
> But it appears the Coordinator will load the data first from the other historicals that were decommissioned before dropping the existing data that is now out of the new retention period:
This is also a known issue proposed in #12881 and being addressed in #13197 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] pchang388 commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
pchang388 commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1218442629
Another issue we noticed during this week of infrastructure issues. When we bring down the historicals from one site, the others get their data loaded. We also swtiched retention period to a lower number (365 -> 90 days) to not run out of disks on the working historicals.
But it appears the Coordinator will load the data first from the other historicals that were decommissioned before dropping the existing data that is now out of the new retention period:
```
2022-08-17T20:01:01,893 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Skipping replica assignment for tier [_default_tier]
2022-08-17T20:01:01,893 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Loading in progress, skipping drop until loading is complete
```
This means that the historicals will fill up first before the drop for retention goes through since it tries to load the data first. Is there any way we can avoid this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1315412876
@pchang388 @danprince1 - You could check out https://github.com/apache/druid/pull/13367 to see if that will help with the issues you are facing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] pchang388 commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
pchang388 commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1219920040
For the original problem, here's an example showing a rebalancing going on after historicals are brought back up in this case:
Task example shows exceeding the default time and failing due to unable to handoff during segment rebalancing
```
2022-08-18T19:39:17,730 INFO [[index_kafka_REDACT_72177abfbb559eb_anenendi]-publish] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Saved sequence metadata to disk: []
2022-08-18T19:39:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:40:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:41:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:42:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:43:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:44:54,138 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:45:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:46:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:47:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:48:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:49:54,135 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:50:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:51:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:52:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:53:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:54:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:55:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:56:54,371 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:57:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:58:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:59:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:00:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:01:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:02:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:03:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:04:54,135 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:05:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:06:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:07:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:08:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:09:00,332 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on issue #12898: Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
Posted by GitBox <gi...@apache.org>.
kfaraz commented on issue #12898:
URL: https://github.com/apache/druid/issues/12898#issuecomment-1315454052
@danprince1 , @pchang388 , thanks for reporting the issue.
There are a set of segment loading changes in #13197 that are currently in review. That PR addresses this and several other issues with segment loading and balancing.
That said, the original issue described here should not really be a problem since newer segments are already prioritized over older segments while loading even in the current code.
References:
https://github.com/apache/druid/blob/309cae7b65e65628f3b277808717a0c7e0bdaccf/server/src/main/java/org/apache/druid/server/coordinator/HttpLoadQueuePeon.java#L86
https://github.com/apache/druid/blob/309cae7b65e65628f3b277808717a0c7e0bdaccf/server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinatorRuntimeParams.java#L52
My guess is that the prioritization is not the issue, the root cause is perhaps that the coordinator is just trying to load everything that is missing and taking very long to do that. If the coordinator gets stuck in a cycle, it doesn't know of new segments that the tasks are publishing and fails to make them available quickly. This problem is being addressed in the mentioned set of changes.
Also, as @abhishekagarwal87 mentioned, #13367 is meant to speed up assignments. Since it is a coordinator dynamic config, you can safely enable and disable it during coordinator operations. The suggested approach would be to enable it before spawning a new historical so that assignments happen quickly.
Once normal operation has resumed, you can choose to disable the round-robin or let it run if your historicals don't seem too out-of-balance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org