You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by GitBox <gi...@apache.org> on 2020/04/30 18:22:38 UTC
[GitHub] [systemml] phaniarnab opened a new pull request #905: [SYSTEMDS-335] Weighted eviction policy
phaniarnab opened a new pull request #905:
URL: https://github.com/apache/systemml/pull/905
This patch contains a new eviction policy for lineage cache. A min-heap
based priority queue over a function of computation time and
size is maintained to define the order of evictions.The idea is to evict
large matrices, which take little time to recompute. This weighted
scheme significantly reduces the number of evictions (including disk spilling).
This patch also refactors the LineageCache class to hide the eviction
policy related maintenance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemml] mboehm7 commented on pull request #905: [SYSTEMDS-335] Weighted eviction policy
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #905:
URL: https://github.com/apache/systemml/pull/905#issuecomment-623123071
LGTM. Thanks for the patch @phaniarnab - I think this is a great direction. During the merge, I fixed the conflicts and simplified `LineageCacheEviction.isSparse` to get the sparsity flag from the actual object not an estimate over the objects metadata. Down the road, it would be good to further generalize the different eviction policies to better separate the different code paths (otherwise, it would get complex and convoluted very quickly) - I left a TODO in `makeSpace`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org