[GitHub] [spark] attilapiros commented on pull request #35559: [SPARK-33206][CORE] Fix shuffle index cache weight calculation for small index files

[GitBox <gi...@apache.org>]

attilapiros commented on pull request #35559:
URL: https://github.com/apache/spark/pull/35559#issuecomment-1048123958


   I have extended the description:
   > But we can go further and get rid of java.io.File completely and store the ShuffleIndexInformation for the file path.
   This way not only the cache size estimate is improved but the its size is decreased as well.
   Here the path size is not counted into the cache size as that string is interned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org