[jira] [Created] (SPARK-38867) Avoid OOM when bufferedPlan has a lot of duplicate keys in SortMergeJoin codegen

["mcdull_zhang (Jira)" <ji...@apache.org>]

mcdull_zhang created SPARK-38867:
------------------------------------

             Summary: Avoid OOM when bufferedPlan has a lot of duplicate keys in SortMergeJoin codegen
                 Key: SPARK-38867
                 URL: https://issues.apache.org/jira/browse/SPARK-38867
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.1
            Reporter: mcdull_zhang


WholeStageCodegenExec is wrapped in BufferedRowIterator.

BufferedRowIterator uses a LinkedList to hold the output of WholeStageCodegenExec.

When the parent of SortMergeJoin cannot codegen, SortMergeJoin needs to append the output to this LinkedList.

SortMergeJoin processes a record in streamedPlan each time. If all records in bufferedPlan can match this record, all records in bufferedPlan will be saved in LinkedList, resulting in OOM.

The above situation is very common in our internal use, so it is best to add a configuration to the codegen code. If there are enough pieces in the LinkedList, stop SortMergeJoin and let the parent consume it first.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org