You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "itschrispeck (via GitHub)" <gi...@apache.org> on 2023/09/12 06:33:18 UTC
[GitHub] [pinot] itschrispeck opened a new issue, #11573: Local temp files are not always cleaned up
itschrispeck opened a new issue, #11573:
URL: https://github.com/apache/pinot/issues/11573
Over time the temporary .tar.gz files accumulate on servers' local disk, requiring manual cleanup. This could be improved with a periodic task to ensure leftover files are removed.
Sample accumulation over a couple months:
```
[docker@<...>/dataDir/<table_name>_REALTIME #]l
consumers/ <table_name>__29__760__20230813T2316Z.tar.gz <table_name>__5__1902__20230820T0323Z.tar.gz
<table_name>__11__1180__20230816T0319Z.tar.gz <table_name>__31__1891__20230820T0321Z.tar.gz <table_name>__53__1105__20230723T0431Z.tar.gz
<table_name>__1__11983__20230715T0116Z.tar.gz <table_name>__31__639__20230813T0705Z.tar.gz <table_name>__53__2025__20230820T1927Z.tar.gz
<table_name>__1__12014__20230715T0839Z.tar.gz <table_name>__33__1111__20230723T0150Z.tar.gz <table_name>__53__906__20230720T1949Z.tar.gz
<table_name>__1__12730__20230722T0925Z.tar.gz <table_name>__33__1263__20230724T1657Z.tar.gz <table_name>__57__1014__20230815T0707Z.tar.gz
<table_name>__1__1625__20230818T1428Z.tar.gz <table_name>__33__1557__20230818T0600Z.tar.gz <table_name>__57__1311__20230816T2142Z.tar.gz
<table_name>__13__1943__20230820T0952Z.tar.gz <table_name>__33__242__20230811T0203Z.tar.gz <table_name>__57__317__20230715T0121Z.tar.gz
<table_name>__13__763__20230813T2304Z.tar.gz <table_name>__33__995__20230815T0357Z.tar.gz <table_name>__57__762__20230813T2315Z.tar.gz
<table_name>__15__1867__20230819T2226Z.tar.gz <table_name>__35__703__20230813T1444Z.tar.gz <table_name>__5__9398__20230707T2025Z.tar.gz
<table_name>__17__244__20230811T0232Z.tar.gz <table_name>__35__713__20230813T1601Z.tar.gz <table_name>__5__9582__20230708T0259Z/
<table_name>__17__443__20230716T0333Z.tar.gz <table_name>__41__1135__20230815T2218Z.tar.gz <table_name>__5__9583__20230708T0301Z/
<table_name>__17__480__20230716T1042Z.tar.gz <table_name>__41__1565__20230818T0754Z.tar.gz <table_name>__5__9584__20230708T0302Z/
<table_name>__17__615__20230717T1808Z.tar.gz <table_name>__41__914__20230814T1813Z.tar.gz <table_name>__59__990__20230815T0358Z.tar.gz
<table_name>__1__769__20230813T2313Z.tar.gz <table_name>__43__1458__20230817T1614Z.tar.gz <table_name>__61__1031__20230722T0821Z.tar.gz
<table_name>__19__237__20230811T0200Z.tar.gz <table_name>__43__1976__20230820T1304Z.tar.gz <table_name>__61__108__20230712T2332Z.tar.gz
<table_name>__1__9434__20230707T1411Z/ <table_name>__45__122__20230712T2323Z.tar.gz <table_name>__61__1881__20230820T0033Z.tar.gz
<table_name>__1__9546__20230707T1919Z/ <table_name>__45__1458__20230817T1618Z/ <table_name>__61__359__20230811T1710Z.tar.gz
<table_name>__19__659__20230813T1001Z.tar.gz <table_name>__45__1458__20230817T1618Z.tar.gz <table_name>__61__666__20230813T1013Z.tar.gz
<table_name>__1__9840__20230708T2244Z.tar.gz <table_name>__47__1141__20230815T2218Z.tar.gz <table_name>__63__1903__20230820T0331Z.tar.gz
<table_name>__1__9899__20230709T0025Z/ <table_name>__47__1458__20230817T1613Z.tar.gz <table_name>__63__2183__20230821T1552Z.tar.gz
<table_name>__21__1245__20230724T1421Z.tar.gz <table_name>__47__1848__20230819T2002Z.tar.gz <table_name>__7__798__20230814T0324Z.tar.gz
<table_name>__21__418__20230715T2322Z.tar.gz <table_name>__47__242__20230811T0205Z.tar.gz <table_name>__9__1654__20230818T1839Z.tar.gz
<table_name>__23__565__20230812T2025Z.tar.gz <table_name>__49__1244__20230724T1656Z.tar.gz <table_name>__9__243__20230811T0212Z.tar.gz
<table_name>__25__1568__20230818T0803Z.tar.gz <table_name>__49__131__20230712T2337Z.tar.gz <table_name>__9__445__20230715T2043Z.tar.gz
<table_name>__25__1894__20230820T0320Z.tar.gz <table_name>__49__1622__20230818T1425Z.tar.gz segment-stats.ser
<table_name>__25__240__20230811T0207Z.tar.gz <table_name>__5__12051__20230712T2046Z.tar.gz _tmp/
<table_name>__29__157__20230713T1035Z.tar.gz <table_name>__5__12076__20230713T0023Z.tar.gz tmp/
<table_name>__29__1588__20230818T1133Z.tar.gz <table_name>__51__700__20230813T1436Z.tar.gz
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Local temp files are not always cleaned up [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11573:
URL: https://github.com/apache/pinot/issues/11573#issuecomment-5824545912
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11573: Local temp files are not always cleaned up
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11573:
URL: https://github.com/apache/pinot/issues/11573#issuecomment-1718604987
Wondering how do we get these orphan segments? Is this caused by peer download failures?
If so, should we put them into a separate folder, and when server starts, just remove all data in the folder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Local temp files are not always cleaned up [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11573:
URL: https://github.com/apache/pinot/issues/11573#issuecomment-6430554675
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Local temp files are not always cleaned up [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11573: Local temp files are not always cleaned up
URL: https://github.com/apache/pinot/issues/11573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org