You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Holden Karau (JIRA)" <ji...@apache.org> on 2019/05/10 19:03:00 UTC

[jira] [Commented] (SPARK-27660) Allow PySpark toLocalIterator to pre-fetch data

    [ https://issues.apache.org/jira/browse/SPARK-27660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837529#comment-16837529 ] 

Holden Karau commented on SPARK-27660:
--------------------------------------

Which is the this an issue duplicate off?

> Allow PySpark toLocalIterator to pre-fetch data
> -----------------------------------------------
>
>                 Key: SPARK-27660
>                 URL: https://issues.apache.org/jira/browse/SPARK-27660
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 3.0.0
>            Reporter: Bryan Cutler
>            Priority: Minor
>
> In SPARK-23961, data was no longer prefetched so that the local iterator could close cleanly in the case of not consuming all of the data. If the user intends to iterate over all elements, then prefetching data could bring back any lost performance. We would need to run some tests to see if the performance is worth it due to additional complexity and extra usage of memory. The option to prefetch could adding as a user conf, and it's possible this could improve the Scala toLocalIterator also.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org