You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "sajjad-moradi (via GitHub)" <gi...@apache.org> on 2023/02/07 02:58:11 UTC
[GitHub] [pinot] sajjad-moradi opened a new issue, #10237: Disallow consuming segment deletion
sajjad-moradi opened a new issue, #10237:
URL: https://github.com/apache/pinot/issues/10237
Recently we had an incident that some consuming segments were deleted by an operator by mistake.
Now that pause/resume are in place, there's no use case for intentionally deleting a consuming segment. In this task, the segment DELETE endpoint needs to be modified to check if the provided segment name is a consuming segment and if so, fail the request with proper error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on issue #10237: Disallow consuming segment deletion
Posted by "PrachiKhobragade (via GitHub)" <gi...@apache.org>.
PrachiKhobragade commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1420134026
I can take it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10237: Disallow consuming segment deletion
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1430211694
Two things:
- We are proposing to prevent deletion of segments in CONSUMING state. ERROR state is not included in this
- Can we change PAUSE to include error segments (optionally)? All it would do is to simply remove the segment, and change the IDEALSTATE to make sure that it is consistent with missing CONSUMING segments. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10237: Disallow consuming segment deletion
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1428927397
>
Hi @Jackie-Jiang what is the use case to delete consuming segment (instead of using the PAUSE API)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10237: Disallow consuming segment deletion
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1421535629
I don't think we should prevent user from deleting a consuming segment because it is a valid operation, and can help when a table run into some weird state. @sajjad-moradi Can you explain the problem you run into when the consuming segment is deleted? We already added an API to explicitly create the consuming segment, and if necessary we can integrate it into the periodic task to automatically create it when none exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10237: Disallow consuming segment deletion
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1435435011
Fair comment on the EV check. I was thinking idealstate check for OFFLINE segments (which is set by us).
Bottom line: We have now defined a valid state that says that if CONSUMING segments are not present, then the idealstate should include the PAUSED bit set. This allows us to reason about the events that may have transpired easily. Another valid state is that consuming segments are present, (maybe in OFFLINE or CONSUMING state) in which case the PAUSED bit is _not_ set in the idealstate.
By allowing the deletion of CONSUMING segments, we are now violating the valid states that IDEALSTATE can be in. IMO this is not a good idea.
If the consuming segments are in ERROR state, and the operator issues the pause command, it should effectively delete the segment, no? We have error to offline and error to dropped state transitions iirc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Disallow consuming segment deletion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10237: Disallow consuming segment deletion
URL: https://github.com/apache/pinot/issues/10237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10237: Disallow consuming segment deletion
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1430202055
For example, there is some malformed data from upstream, which is causing consuming segment to run into ERROR state. Pausing might not help because segment metadata manager is already error'ed out. In order to manually skip the bad data, we want to delete the consuming segment and re-create the consuming segment with the latest offset.
Adding an extra argument to explicitly delete consuming segment sounds good to me. We might want to reject the deletion when the flag is set but the segment is not consuming segment to prevent user from always setting the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10237: Disallow consuming segment deletion
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1431848472
> * We are proposing to prevent deletion of segments in CONSUMING state. ERROR state is not included in this
I don't follow this. Are you saying you want to check EV to decide whether a segment can be deleted? That doesn't sound correct to me.
> * Can we change PAUSE to include error segments (optionally)? All it would do is to simply remove the segment, and change the IDEALSTATE to make sure that it is consistent with missing CONSUMING segments. wdyt?
This proposal is essentially making PAUSE to delete the consuming segment. IMO this behavior is not intuitive to the user. I still suggest allowing the primitive operation of deleting a segment. To prevent mis-operation, adding a flag is the preferred solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10237: Disallow consuming segment deletion
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1435727442
@Jackie-Jiang this blog post explains how pause/resume can solve the exact problem that you are referencing:
https://startree.ai/blog/apache-pinot-0-11-pausing-real-time-ingestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #10237: Disallow consuming segment deletion
Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on issue #10237:
URL: https://github.com/apache/pinot/issues/10237#issuecomment-1428913999
@Jackie-Jiang In our clusters, whenever a consuming segment was deleted, it was a result of a user error. So we prefer to not delete consuming segments. That being said, I see your point of using delete endpoint for deleting some consuming segments being in weird state.
What do you think of adding an extra flag indicating force-deleting a consuming segment?
```
DELETE /segments/{tableName}/{segmentName}
returns 400 if the segment is in consuming state, with error saying “{segmentName} is in consuming state. If you need to delete the consuming segment, provide ‘forceDeleteConsumingSegment’ parameter”.
DELETE /segments/{tableName}/{segmentName}?forceDeleteConsumingSegment=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org