You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/28 21:28:05 UTC
[GitHub] [pinot] walterddr opened a new issue #8091: table creation shouldn't be allowed if table delete is happening async in background
walterddr opened a new issue #8091:
URL: https://github.com/apache/pinot/issues/8091
Currently table is deleted first and the segments were deleted async behind the scene.
however if the controller died during the async deletion and got restarted. User can call the API to create an exact same name table without any issue. This should not be allowed since there's still left-over segments.
I am not sure what's the right way to fix it. but some thoughts:
1. create a soft delete state for a table marking it as deleting, and not allow same name table to be recreated until the segment deletion is done.
2. check the property store as well as table config store to make sure no left over segments exist, otherwise reject the table config creation request.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr removed a comment on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
walterddr removed a comment on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1024663985
sweet. thanks for referring to #7711. is it a duplicate entirely? i think we can close this one if it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ksnijjer commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
ksnijjer commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1024699296
For the user issue that we saw, table information was gone from EV/IS however entries were still left(controller seemed to die/restart in the middle) in ZK>PropertyStore(Table def. and Segments). Now table creation was still going through returning a 200 response but table didn't get created, so only checking IS/EV to allow table creation won't help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1026054424
Observation
===
The issue is whether EXTERNAL VIEW for a table goes away before or after the PropertyStore remove entries for the associated segments.
Upon checking the code in https://github.com/apache/pinot/blob/dd73ee7de1d8e8439e71b34d24627db48e06990e/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java#L1624 the order for deleting offline table is
* DropResource --> RemoveSegments --> RemovePropertyStore --> RemoveTableConfig --> RemoveInstanceAssignment
and in https://github.com/apache/pinot/blob/dd73ee7de1d8e8439e71b34d24627db48e06990e/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java#L1259
* AddResource --> AddTableConfig --> AddInstanceAssignment
ProblemStatement
===
So the problem I think is
1. addTable does't check segment info and see if there's already existing segments for this table (with the same name i asusme)
2. deleteTable shouldn't delete the helix resource first before all the segments are remove (which is the only thing addTable checks for before raising TableAlreadyExistException)
Proposal
===
Looking at the above 2 problem, my proposal is to either
1. check for segment in property store as well as Helix resource for duplicate table name
2. deleteTable should not delete the helix resource before all the table config and the segment info in property store were deleted.
Thoughts?
Reference
===
relates to #8092. Similar issue on the ordering - whether deep store file were deleted **before / after** property store segment deletion;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1026127214
I have the same observation as @Jackie-Jiang and also lean towards the 1st solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
walterddr commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1024663985
sweet. thanks for referring to #7711. is it a duplicate entirely? i think we can close this one if it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #8091:
URL: https://github.com/apache/pinot/issues/8091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1024660259
Related to Issue #7711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1024661280
During deletion:
- Start the deletion process and wait until the table goes away from EXTRERNALVIEW
During table creation:
- Verify that the table does not exist in IDEALSTATE or EXTERLAVIEW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8091: table creation shouldn't be allowed if table delete is happening async in background
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8091:
URL: https://github.com/apache/pinot/issues/8091#issuecomment-1026099184
When creating the table, we should check all the related entries for the table is already removed. Per the sequence of entry deletion, we should check table config, instance partitions, segment lineage, task related metadata.
EV should not matter because Helix is able to handle it properly when the new IS is created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org