[GitHub] spark issue #16909: [SPARK-13450] Introduce ExternalAppendOnlyUnsafeRowArray...

[sheperdh <gi...@git.apache.org>]

Github user sheperdh commented on the issue:

    https://github.com/apache/spark/pull/16909
  
    I have one question regarding this change.
    We are doing some tests using spark-sql.
    In our SQL, we have used "Full Outer Join", we found it's quite easy to hit OOM when there is large dataset.
    After analyzed the heap dump, we found it's because for "Full Outer Join", we haven't used the ExternalAppendOnlyUnsafeRowArray. 
    I am not sure whether there is any special reason why Spark-SQL doesn't optimize "Full Outer Join".
    
    @tejasapatil 
    Can you let me know why you don't want to support "Full Outer Join"?
    
    thanks,
    Sheperd


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org