You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/04/21 22:37:58 UTC

[jira] [Commented] (SPARK-4621) when sort- based shuffle, Cache recently finished shuffle index can reduce indexFile's io

    [ https://issues.apache.org/jira/browse/SPARK-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505703#comment-14505703 ] 

Josh Rosen commented on SPARK-4621:
-----------------------------------

I have a partially-written patch for this issue, but I still need to determine whether this buys us significant performance benefits.  The small index file might be likely to remain in the OS buffer cache, so maybe the cost of the current approach isn't that high.  Would be good to measure this, though.

> when sort- based shuffle, Cache recently finished shuffle index can reduce indexFile's io
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-4621
>                 URL: https://issues.apache.org/jira/browse/SPARK-4621
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>            Reporter: Lianhui Wang
>            Priority: Minor
>
> in IndexShuffleBlockManager, we can use LRUCache to store recently finished shuffle index and that can reduce indexFile's io.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org