[GitHub] [superset] matschaffer-roblox commented on issue #8155: log retention / deletion

["matschaffer-roblox (via GitHub)" <gi...@apache.org>]

matschaffer-roblox commented on issue #8155:
URL: https://github.com/apache/superset/issues/8155#issuecomment-1424957043

   Incase it helps anyone I ended up just doing this after taking a backup:
   
   ```
   while true; do
     TZ=UTC date
     echo starting batch
     mysql (connection info) -e 'DELETE FROM logs WHERE `dttm` < (NOW() - INTERVAL 90 DAY) LIMIT 1000000;'
     echo sleeping 5min
     sleep 300
   done
   ```
   
   The downside is that this doesn't really _free_ space but it should (I think) allow new logs to reuse the blocks left open from the deletions.
   
   For our DB each deletion of 1 million took about 1 minute to complete. I tested a select & insert while running and they worked fine, so I don't think this will cause locking issues since nothing (afaik) is attempting to update rows in the logs table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org