You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/10/27 07:35:53 UTC
[GitHub] [incubator-uniffle] jerqi opened a new issue, #282: [FEATURE] Add a period local storage cleaner thread
jerqi opened a new issue, #282:
URL: https://github.com/apache/incubator-uniffle/issues/282
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Because we don't use strict lock strategy, we will delete some shuffle data normally. If we use strict lock strategy, the performance will be bad. So we want to add a period local storage cleaner thread. The thread examine which application is in the disk and judge whether the app exists the memory data structure `ShuffleTaskManager#shuffleTaskInfos`. If memory data structure don't exist, we will delete the data in the disk.
### Motivation
Reduce the garbage data. The data won't be deleted unless the shuffle server is restarted.
### Describe the solution
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #282:
URL: https://github.com/apache/incubator-uniffle/issues/282#issuecomment-1319986251
@sfwang218 Will you continue this issue? If you need some help, I'm pleased to give you a hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] sfwang218 commented on issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
sfwang218 commented on issue #282:
URL: https://github.com/apache/incubator-uniffle/issues/282#issuecomment-1322080580
> @sfwang218 Will you continue this issue? If you need some help, I'm pleased to give you a hand.
Sorry, I'm busy recently, Could you give me some suggestion? I will do it this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] sfwang218 commented on issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
sfwang218 commented on issue #282:
URL: https://github.com/apache/incubator-uniffle/issues/282#issuecomment-1323692847
> We can modify the ShuffleTaskManger and LocalStorageManager to implement this function. For ShuffleTaskManager, you can see the method `removeResourcesByShuffleIds`, we can refer to it and then add a daemon thread. For LocalStorageManager, you can add a interface `checkLeakShuffleData(Collection<String> appIds)` for StorageManager. LocalStorageManager implement this interface.
I think this way does not work, because the appId is a already removed from memory data structure ShuffleTaskManager#shuffleTaskInfos, and we can only parse the appid from the Storage file directory,
and then Compare with ShuffleTaskManager#shuffleTaskInfos.
I'm doing it in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] sfwang218 commented on issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
sfwang218 commented on issue #282:
URL: https://github.com/apache/incubator-uniffle/issues/282#issuecomment-1294329826
Could you assign this issue to me? I want to try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
jerqi closed issue #282: [FEATURE] Add a period local storage cleaner thread
URL: https://github.com/apache/incubator-uniffle/issues/282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #282: [FEATURE] Add a period local storage cleaner thread
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #282:
URL: https://github.com/apache/incubator-uniffle/issues/282#issuecomment-1322205957
We can modify the ShuffleTaskManger and LocalStorageManager to implement this function.
For ShuffleTaskManager, you can see the method `removeResourcesByShuffleIds`, we can refer to it and then add a daemon thread.
For LocalStorageManager, you can add a interface `checkLeakShuffleData(Collection<String> appIds)` for StorageManager. LocalStorageManager implement this interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org