[GitHub] [incubator-uniffle] smallzhongfeng commented on pull request #168: [Improvement] Should match from pathToStorages when appId does not exist in appIdToStorages

[GitBox <gi...@apache.org>]

smallzhongfeng commented on PR #168:
URL: https://github.com/apache/incubator-uniffle/pull/168#issuecomment-1221920900

   Because we found the log `can't find HDFS storage for appid`, it means that `appIdToStorages` has removed the cache of this app. Because `removereSources` was called only once, but from the HDFS audit log, it can be seen that the shuffle file was deleted at 18:00:55, but was rewritten at 18:00:57, resulting in the loss of the appId of the cache of `appIdToStorages`, However, the cached appId of `shuffleTaskInfos` is still there and cannot be removed. Therefore, this exception occurs when `removeResources` is scheduled again. Therefore, this PR is mainly used to solve the problem that the path cannot be deleted after the cache of appid by appidtostorages expires.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org