You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Pankaj260100 (via GitHub)" <gi...@apache.org> on 2024/04/16 08:31:02 UTC
[I] Ingestion lag because of multiple checkpoints. (druid)
Pankaj260100 opened a new issue, #16293:
URL: https://github.com/apache/druid/issues/16293
### Description
We see multiple checkpoints gets triggered back-to-back from a ingestion task when hit the maxTotalRows in one of the task replica and ingestion replica is greater than equal to 2.
These multiple checkpoints can lead to 2 scenarios:
- **scenario 1**: Ingestion gets throttled becuase of multiple persists
- **scenario 2**: Ingestion task got killed by overlord, becuase it failed to pause in a timely manner when overlord was handling checkpoint requests.
**Why we do not see this when ingestion replica is less than 2?**
- Let's take an example: Task A, ingesting from 2 partitions(x, y) and currently x=100 and y =50 offset, and task A hits maxTotalRows( let suppose it is set to 150)
T1(x = 100, y = 50) ------- Hit row limit, issue a checkpoint request and set the endoffset to (100, 50) ------ this task goes to `maybePersistAndPublishSequences` and publish the segment as it is created and update the maxTotalRows-(number of rows published) And start ingesting again into the new sequence.
**Why we see this with ingestion replica is greater than equal to 2?**
- Ex: Task A and its replica Task B, ingesting from 2 partitions. MaxTotalRow is set to 150. (Task B might have handoff the earlier segment that's why it ahead and still do not hit the maxTotalRows).
```
CurrentTotalRows= 150 for task A
Task A(x1= 100 and y1 = 50) Task B(x1 = 200 and y1= 75)
| |
| |
| |
hit MaxTotalRows/Send Checkpoint |
|
| both tasks got paused |
| |
|----------- set endoffset(200,75) ---------------------------- |
| |
| |
TaskA again start ingesting in older sequence. Task B can go ahead and publish the segment as it has already reached
| endoffset for both partition
| |
| |
Task A (x1= 150, y1 = 75) |
for partition y1 it will start ingesting into new sequence |
as it has reached endoffset 75 for y1. |
curr Total Rows is 225(> 150) |
new sequence is not checkpointed **Scenario2:** Sometime when Task A issued a checkpoint and
So, Task A again sends a checkpoint request overlord askTask A and Task B to pause. But task B busy
| publishing the segment to s3. It won't respond to pause
| request from overlord and after exhausting all retries
overlord ask the worker to kill this task. And it can go into very bad
| state if this task has published the segment and got killed, we
| have seen all the Subsequent task failed becuase of metadata mismatch.
|
| // **scenario1** The task gets paused, it will again set the endoffset
| and start publishing/ ingesting into new seq. But in this case we
| see more than 1 persist throttled the ingestion
|
|
|
After checkpointing if it again reaches the endoffset for
partition y1 before partition x1 reaches 200 it will again
issue a new checkpoint request as currentTotalRows will remain high.
we have not published the segment as partition x1 is behind.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingestion lag because of multiple checkpoints. (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16293:
URL: https://github.com/apache/druid/issues/16293#issuecomment-2608559552
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingestion lag because of multiple checkpoints. (druid)
Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on issue #16293:
URL: https://github.com/apache/druid/issues/16293#issuecomment-2061942855
In short, If the task is consuming from multiple partitions and, for some reason, the consumption from multiple kafka partitions is different, the task will continue to hold the older segment. This causes multiple checkpoint requests and the above issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingestion lag because of multiple checkpoints. (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16293:
URL: https://github.com/apache/druid/issues/16293#issuecomment-2670076377
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Ingestion lag because of multiple checkpoints. (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16293: Ingestion lag because of multiple checkpoints.
URL: https://github.com/apache/druid/issues/16293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org