[GitHub] [pinot] diogobaeder opened a new issue #7711: Table synchronous dropping

[GitBox <gi...@apache.org>]

diogobaeder opened a new issue #7711:
URL: https://github.com/apache/pinot/issues/7711


   Hey there folks,
   
   Currently it's too hard to run any integration tests between my Python-based project and Apache Pinot, because it's really hard to have each test running starting from a deterministic state regarding Pinot. I already tried deleting all segments, tried dropping the table and recreating it, and none of these approaches work well enough - deleting segments requires me to manually recreate new ones afterwards, which I haven't found an easy way to do, for example. When it comes to deleting tables, I found it was a bit odd that a table can be deleted but this is not done right away, and if I recreate a new and equal table right away it comes up with the same segments as before.
   
   Therefore I think it would be nice if there could be an endpoint that could be used for synchronously dropping a table, while ensuring it gets fully cleared off of Pinot before responding - this way it would be possible to reliably dropping and then recreating the table for each integration test I have. It doesn't have to be the same endpoint that already exists for dropping a table, but it should be something that guarantees a deterministic state after responding to the client-side.
   
   I thought about asking for table truncation, but I noticed there's another ticket for handling that, and it's tricky to implement according to the people discussing that, so I guess synchronously dropping a table would be easier to implement as a workaround.
   
   What do you think? Can this be implemented? Perhaps in time for the 0.9.0 release?
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mcvsubbu commented on issue #7711: Table synchronous dropping

[GitBox <gi...@apache.org>]

mcvsubbu commented on issue #7711:
URL: https://github.com/apache/pinot/issues/7711#issuecomment-962249450


   For testing purpose, you can also consider killing all zk, and pinot processes and restarting them (you will need to start zk process without saving to persistent store).
   
   For production use case, implementing a table delete api that is idempotent is possible I believe. Essentially, the controller API needs to do the following steps (not necessarily in this sequence):
   - disable table, remove all metadata
   - Remove all segments from idealstate
   - await externalview to match idealstate.
   - drop the helix resource for the table
   
   Things to consider:
   - A controller could get restarted between the above steps, and the client could re-issue the API.
   - the (http) client (of the controller api) could get disconnected, re-connect to a different controller and do the same operation. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] diogobaeder commented on issue #7711: Table synchronous dropping

[GitBox <gi...@apache.org>]

diogobaeder commented on issue #7711:
URL: https://github.com/apache/pinot/issues/7711#issuecomment-962293150


   Hey there @mcvsubbu !
   
   Thanks for the suggestion, but that's not a viable workaround for my project; The thing is, not only we run those tests locally, but we also run them on GitLab-CI, where we spawn services as docker containers using GitLab-provided tooling and then run the tests, so we'll need to have those components running during the whole test suite.
   
   I think the idea of deleting the table guaranteeing it's an idempotent operation sounds good, and it should be expected that it might take a while, and I'm fine with that. And there might be corner cases too, I'm fine with that too, I think starting somewhere and understanding there might be issues during that kind of operation is acceptable, maybe in the future this could be improved, but at least for automated tests it would still be a very beneficial feature to have.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org