You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/08 22:46:03 UTC
[GitHub] [pinot] jtao15 opened a new issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
jtao15 opened a new issue #7413:
URL: https://github.com/apache/pinot/issues/7413
There's a potential race condition issue when we run merge/roll-up task & segment upload tasks (e.g. purge, convertToRawIndex, etc) on the same table (purge task for example):
1. Segment A is picked both by merge/roll-up and purge tasks
2. Merge/roll-up task is finished, and the retention manager tries to clean up segment A and the lineage entry before purge task uploading the purged segment.
3. Purge task uploaded the segment, and we have duplicate data for the table.
There are two potential improvements:
1. Honor segment lineage when scheduling purge tasks to avoid picking segments that will be removed.
2. Existence check + ZK version check when uploading segments by purge tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jtao15 commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
jtao15 commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-917294620
> I think the key to solve this problem is to keep the lineage entry longer and only remove it when the segment is already removed in the previous round of cleanup. Combine this with segment existence check should be good enough to solve this race condition
In extreme case:
1. Segment A is picked for merge/roll-up
2. Before the merge/roll-up task is done, A is picked for purging
3. merge task is finished
4. retention manager removed segment A
5. after 6 hours, retention manager removed the lineage
6. Purge task is done, the purged segment is uploaded and we have duplicate data
So if the purge task run longer than 6 hour, we still have the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-915649144
Does it make sense to have some sort of segment level locks so that the same set of segments are not picked up by multiple tasks? If we can come up with a lock-free algorithms on this, that will be even better.
Alternatively, some config value of not doing merge/rollup if the retention is just N hours/days away (if this is the only race condition).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-917294437
I think we do check the CRC when purge task pushes the segment. If that is not the case, we should add existence and CRC check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-916373241
I think the key to solve this problem is to keep the lineage entry longer and only remove it when the segment is already removed in the previous round of cleanup. Combine this with segment existence check should be good enough to solve this race condition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-918512486
@jtao15 One way to ensure atomicity is to follow the following steps when replacing the segment:
1. Read the existing segment ZK metadata
2a. If not exist, abort
2b. If exist, check CRC
3a. If not match, abort
3b. If match, update segment ZK metadata with ZNode version check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jtao15 commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
jtao15 commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-917290051
> Does it make sense to have some sort of segment level locks so that the same set of segments are not picked up by multiple tasks? If we can come up with a lock-free algorithms on this, that will be even better.
>
> Alternatively, some config value of not doing merge/rollup if the retention is just N hours/days away (if this is the only race condition).
The segment zk metadata version will achieve the segment level lock? If we always check the version when replacing/deleting the segments, only one task will succeed. The other replacing/deleting request can retry later.
I think scheduling part will not cause any issue, but the clean up part of merge/roll-up does. By adding merge/roll-up task, the retention manager now need to clean up merged segment to free the disk space. Say we merge A and B to C. After merging is done, the retention manager will remove A and B (even if they didn't pass the retention threshold, they should be removed since they are not serving queries anymore). If A and B are scheduled with purge task, the race condition can happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #7413:
URL: https://github.com/apache/pinot/issues/7413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang edited a comment on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang edited a comment on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-918512486
@jtao15 One way to ensure atomicity is to follow the following steps when replacing the segment:
1. Read the existing segment ZK metadata
2. If not exist, abort
3. If exist, check CRC
4. If not match, abort
5. If match, update segment ZK metadata with ZNode version check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jtao15 commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks
Posted by GitBox <gi...@apache.org>.
jtao15 commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-917298621
> I think we do check the CRC when purge task pushes the segment. If that is not the case, we should add existence and CRC check
We do check the crc. But the version check is still needed since in extreme cases the segment can be deleted after we check crc since the check&update is not atomic?
I think we can have some sort of flag to indicate the upload request is replacement only (for purge and convertToRawIndex), if there's no existing segment or the segment got deleted before the request complete (zk version mismatch), we just abort the request. @Jackie-Jiang What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org