[GitHub] [druid] jihoonson commented on issue #11297: Using deep storage as intermediate store for shuffle tasks

[GitBox <gi...@apache.org>]

jihoonson commented on issue #11297:
URL: https://github.com/apache/druid/issues/11297#issuecomment-879273572


   >  For the first version, I wonder if it would be ok to suggest the Druid Operator to set up a separate Lifecycle policy rule on the objects in the intermediary path to auto-expire objects for the boundary cases.
   
   @nishantmonu51 I agree that auto cleanup can be developed separately from storing intermediary data in deep storage. Also using auto-expiry seems OK to me as long as it is documented clearly how to set the retention and what could be the impact when it is set wrong. The problem of using auto-expiry is it will be hard to set the retention correctly since the retention should be longer than your longest job which depends on input data and the resource available while the job is running. Perhaps it should be marked as experimental (or alpha could be a better term) until the auto cleanup is developed in druid.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org