You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/17 05:07:54 UTC

[GitHub] [spark] viirya commented on pull request #29772: [SPARK-32900][CORE] Allow UnsafeExternalSorter to spill when there are nulls.

viirya commented on pull request #29772:
URL: https://github.com/apache/spark/pull/29772#issuecomment-693863796


   > Currently, Spark determines whether UnsafeExternalSorter.SpillableIterator has spilled already by checking whether upstream is an instance of UnsafeInMemorySorter.SortedIterator
   
   Can we update the description a bit?
   
   This is reading like Spark thinks `UnsafeExternalSorter.SpillableIterator` has spilled already if `upstream` is `UnsafeInMemorySorter.SortedIterator`. But it actually is Spark thinks `UnsafeExternalSorter.SpillableIterator` has spilled already if `upstream` is not `UnsafeInMemorySorter.SortedIterator`, right?
   
   ```scala
   if (!(upstream instanceof UnsafeInMemorySorter.SortedIterator && nextUpstream == null
     && numRecords > 0)) {
     return 0L;
   }
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org