[jira] [Created] (SPARK-36324) Replace revertPartialWritesAndClose with close in ExternalSorter.spillMemoryIteratorToDisk and ExternalAppendOnlyMap.spillMemoryIteratorToDisk

["Yang Jie (Jira)" <ji...@apache.org>]

Yang Jie created SPARK-36324:
--------------------------------

             Summary: Replace revertPartialWritesAndClose with close in ExternalSorter.spillMemoryIteratorToDisk and ExternalAppendOnlyMap.spillMemoryIteratorToDisk 
                 Key: SPARK-36324
                 URL: https://issues.apache.org/jira/browse/SPARK-36324
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.3.0
            Reporter: Yang Jie


ExternalAppendOnlyMap.ExternalAppendOnlyMap call  `revertPartialWritesAndClose` method when `objectsWritten == 0` as follows:
{code:java}
try {
  while (inMemoryIterator.hasNext) {
    ...
    if (objectsWritten == serializerBatchSize) {
      flush()
    }
  }
  if (objectsWritten > 0) {
    flush()
    writer.close()
  } else {
    writer.revertPartialWritesAndClose()
  }
  success = true
} finally {
  ...
}{code}
 

writer.revertPartialWritesAndClose() can replace with writer.close to reduce a set of file operations includes open, truncate and close.

 

A similar situation exists for ExternalSorter

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org