You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/26 15:14:50 UTC
[GitHub] [pinot] walterddr opened a new issue #8072: Support instant deletion
walterddr opened a new issue #8072:
URL: https://github.com/apache/pinot/issues/8072
In our current `SegmentDeletionManager`, deleting a segment or all segments in deep store result in a move from active segment file to deleted_ prefix segment files; then `RetentionManager` checks and deletes these files after a RETENTION PERIOD default to 7 days.
In cloud object stored backed PinotFS like S3, `move` is essentially a copy-and-delete operation and thus could cause huge overhead during the delete operation if the entire table contains too many segments.
The proposal here is to support instant delete without going through RetentionManager (see #8069 as implementation).
The question are:
1. whether we want to support instant delete other than a total-table-deletion scenario?
2. should we provide the user interface as a tableConfig, a REST API parameter, or simply a default only when deleting the entire table (without changing the API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022647715
I still don't understand why the latency is a problem. Is it that the background task takes too long to complete ? Or, is it that each I/O to S3 costs, and we want to minimize the number of I/Os (at the cost of not having a backup, of course)? Or, is it both?
Assuming it is both, the best way to solve this problem as of now will be to
- Set the retention days to 0 (instead of default 7) AND
- Change the code so that deletion manager deletes it right away if the retention time is set to 0.
This is a cluster-wide behavior. So, if you want to change this on a per table basis, we should have a table config (that overrides the cluster config) with a different retention time (which can be 0 to indicate immediate deletion).
A change to the API is not the best way to solve this IMO. You could have clients make a mistake while deletion, causing some segments to be retained and others to be deleted. Better to enforce uniform behavior across all segments in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022620083
Recreating a table is a more involved problem. What we need to do is to wait for the external view to disappear in the table drop API. I believe there is an issue for it already.
Deletion of segments from the deep store happens right away, it is the wait on externalview that takes time. Hopefully in your cluster, this is a short time.
Deletion via retention manager happens MUCH later and does not interfere with the recreation of a table.
Let us formulate the problem correctly before we start looking for solutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022697265
@mcvsubbu agreed! I was not proposing to change the API to solve the problem (that's why I created this issue instead of continuing the discussion in the previous PR), as the proposal is to "find a way support instantly delete the segment".
I am fine with going through setting the RETENTION setting to 0 instead of 7 approach, as long as I can change the logic (as you mentioned) to instantly delete in SegmentDeletionManager
Regarding the use cases @Jackie-Jiang mentioned, I think we can achieve both scenarios (technically) via using a table-level config that overrides cluster level config as @mcvsubbu mentioned. whether this is convenient for the user or not is a different story and we can leave that out of this discussion.
So to summarized, I will
1. change the logic in SegmentDeletionManager and instantly delete the segment if retention period is set to 0
2. create a tableConfig for retention that overrides cluster level config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022694357
What does "super-expensive" mean? We should not be adding/deleting segments at hundreds per second, so I am just curious here. I think segment reads/downloads will have much more cost (but perhaps that is where the "tiered storage" comes in).
With that, yes, there may be use cases where user wants to drop the segment or segments right away. So, both segment delete and table delete can have the ultimate override, maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022629011
hmm I think I understand your confusion when I give examples here and there. let me rephrase the entire description of the issue and let's restart the discussion from this point forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022668825
I agree we should have both cluster config and table config override for retention time, but we also want to give user the flexibility to delete the segment immediately without moving to a deleted folder through the rest API (note that for some cloud storage, `move` is not supported, and it is implemented as `copy` then `delete` which can be super expensive).
Some scenarios I can think of:
- User pushed a bad segment (or several) and want to delete it
- User wants to delete a table they no longer need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr edited a comment on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr edited a comment on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022629011
hmm I think I understand your confusion when I give examples here and there. let me rephrase the entire description of the issue and let's restart the discussion from this point forward.
[Updated] Hi @mcvsubbu, please let me know if the updated problem statement is clear or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr edited a comment on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr edited a comment on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022574161
-- deleted comment --
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022574161
Hi @mcvsubbu, yes. I think the problem I stated to solve is "this (copy-and-delete) cause huge overhead during the deletion operation". this is both in time and I/O cost.
One example is I wanted to delete and recreate a table, it is not ideal to wait until the background task to finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr edited a comment on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
walterddr edited a comment on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022574161
Hi @mcvsubbu, yes. I think the problem I stated to solve is "this (copy-and-delete) cause huge overhead during the deletion operation". this is both in time and I/O cost.
Event as a background task, one example that caused problem is: I wanted to delete and recreate a table, it is not ideal to wait until the background task to finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8072: Support instant deletion
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8072:
URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022473268
Can you state the problem you want to solve? Is it that the retention of a deleted segment taking up storage cost in the cloud? Or, is it that deletion takes a long time (it is a background task, so I don't see why that should matter much).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org