You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2018/04/01 15:01:00 UTC
[jira] [Updated] (SPARK-23040) BlockStoreShuffleReader's return
Iterator isn't interruptible if aggregator or ordering is specified
[ https://issues.apache.org/jira/browse/SPARK-23040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan updated SPARK-23040:
--------------------------------
Fix Version/s: 2.3.1
> BlockStoreShuffleReader's return Iterator isn't interruptible if aggregator or ordering is specified
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-23040
> URL: https://issues.apache.org/jira/browse/SPARK-23040
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1
> Reporter: Xianjin YE
> Assignee: Xianjin YE
> Priority: Minor
> Fix For: 2.3.1, 2.4.0
>
>
> For example, if ordering is specified, the returned iterator is an CompletionIterator
> {code:java}
> dep.keyOrdering match {
> case Some(keyOrd: Ordering[K]) =>
> // Create an ExternalSorter to sort the data.
> val sorter =
> new ExternalSorter[K, C, C](context, ordering = Some(keyOrd), serializer = dep.serializer)
> sorter.insertAll(aggregatedIter)
> context.taskMetrics().incMemoryBytesSpilled(sorter.memoryBytesSpilled)
> context.taskMetrics().incDiskBytesSpilled(sorter.diskBytesSpilled)
> context.taskMetrics().incPeakExecutionMemory(sorter.peakMemoryUsedBytes)
> CompletionIterator[Product2[K, C], Iterator[Product2[K, C]]](sorter.iterator, sorter.stop())
> case None =>
> aggregatedIter
> }
> {code}
> However the sorter would consume(in sorter.insertAll) the aggregatedIter(which may be interruptible), then creates an iterator which isn't interruptible.
> The problem with this is that Spark task cannot be cancelled due to stage fail(without interruptThread enabled, which is disabled by default), which wasting executor resource.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org