[jira] [Comment Edited] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

["Jianfei Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496283#comment-15496283 ] 

Jianfei Wang edited comment on SPARK-17562 at 9/16/16 1:03 PM:
---------------------------------------------------------------

this func is to revert writes that haven't been committed yet.
If there are exceptions,we will go into finally.
Besides,if there are 0 object written,we needn't to call this to revert,it will do nothing.


was (Author: codlife):
this func is to revert writes that haven't been committed yet.
If there are exception,we will go into finally.
Besides,if there are 0 object written,we needn't to call this to revert,it will do nothing.

> I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-17562
>                 URL: https://issues.apache.org/jira/browse/SPARK-17562
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.0.0
>            Reporter: Jianfei Wang
>            Priority: Trivial
>              Labels: easyfix, performance
>
> In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never be executed, so we can remove them
> {code}
> else  {
> writer.revertPartialWritersAndClose()
> }
> {code}
> the source code is as below:
> {code}
> try {
>       while (inMemoryIterator.hasNext) {
>         val partitionId = inMemoryIterator.nextPartition()
>         require(partitionId >= 0 && partitionId < numPartitions,
>           s"partition Id: ${partitionId} should be in the range [0, ${numPartitions})")
>         inMemoryIterator.writeNext(writer)
>         elementsPerPartition(partitionId) += 1
>         objectsWritten += 1
>         if (objectsWritten == serializerBatchSize) {
>           flush()
>         }
>       }
>       if (objectsWritten > 0) {
>         flush()
>       } else {
>         writer.revertPartialWritesAndClose()
>    }
>       success = true
>     } finally {
>       if (success) {
>         writer.close()
>       } else {
>         // This code path only happens if an exception was thrown above before we set success;
>         // close our stuff and let the exception be thrown further
>         writer.revertPartialWritesAndClose()
>         if (file.exists()) {
>           if (!file.delete()) {
>             logWarning(s"Error deleting ${file}")
>           }
>         }
>       }
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org