You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2020/10/12 10:53:00 UTC
[jira] [Updated] (SPARK-33119) ScalarSubquery should returns the
first two rows to avoid Driver OOM
[ https://issues.apache.org/jira/browse/SPARK-33119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-33119:
--------------------------------
Summary: ScalarSubquery should returns the first two rows to avoid Driver OOM (was: Only return the first two rows to avoid Driver OOM)
> ScalarSubquery should returns the first two rows to avoid Driver OOM
> ---------------------------------------------------------------------
>
> Key: SPARK-33119
> URL: https://issues.apache.org/jira/browse/SPARK-33119
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Yuming Wang
> Priority: Major
>
> {noformat}
> Exception in thread "subquery-2871" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:103)
> at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
> at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1$$anonfun$apply$2.apply(SparkPlan.scala:352)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1$$anonfun$apply$2.apply(SparkPlan.scala:352)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at org.apache.spark.sql.execution.SparkPlan$$anon$1.foreach(SparkPlan.scala:330)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1.apply(SparkPlan.scala:352)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1.apply(SparkPlan.scala:351)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:351)
> at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.executeCollect(AdaptiveSparkPlanExec.scala:274)
> at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1$$anonfun$apply$3.apply(basicPhysicalOperators.scala:830)
> at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1$$anonfun$apply$3.apply(basicPhysicalOperators.scala:827)
> at org.apache.spark.sql.execution.SQLExecution$$anonfun$withExecutionId$1.apply(SQLExecution.scala:132)
> at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:156)
> at org.apache.spark.sql.execution.SQLExecution$.withExecutionId(SQLExecution.scala:129)
> at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1.apply(basicPhysicalOperators.scala:827)
> at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1.apply(basicPhysicalOperators.scala:827)
> at scala.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org