You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2021/07/28 07:30:00 UTC
[jira] [Updated] (SPARK-36324) Replace revertPartialWritesAndClose
with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
[ https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie updated SPARK-36324:
-----------------------------
Summary: Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill (was: Replace revertPartialWritesAndClose with close in ExternalSorter.spillMemoryIteratorToDisk and ExternalAppendOnlyMap.spillMemoryIteratorToDisk )
> Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-36324
> URL: https://issues.apache.org/jira/browse/SPARK-36324
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Yang Jie
> Priority: Minor
>
> ExternalAppendOnlyMap.ExternalAppendOnlyMap call `revertPartialWritesAndClose` method when `objectsWritten == 0` as follows:
> {code:java}
> try {
> while (inMemoryIterator.hasNext) {
> ...
> if (objectsWritten == serializerBatchSize) {
> flush()
> }
> }
> if (objectsWritten > 0) {
> flush()
> writer.close()
> } else {
> writer.revertPartialWritesAndClose()
> }
> success = true
> } finally {
> ...
> }{code}
>
> writer.revertPartialWritesAndClose() can replace with writer.close to reduce a set of file operations includes open, truncate and close.
>
> A similar situation exists for ExternalSorter
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org