You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Leanken.Lin (Jira)" <ji...@apache.org> on 2020/08/14 06:54:00 UTC
[jira] [Created] (SPARK-32615) AQE aggregateMetrics
java.util.NoSuchElementException
Leanken.Lin created SPARK-32615:
-----------------------------------
Summary: AQE aggregateMetrics java.util.NoSuchElementException
Key: SPARK-32615
URL: https://issues.apache.org/jira/browse/SPARK-32615
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Leanken.Lin
Reproduce Step
{code:java}
//代码占位符
sql/test-only org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z "SPARK-32573: Eliminate NAAJ when BuildSide is EmptyHashedRelationWithAllNullKeys"
{code}
{code:java}
//代码占位符
14:40:44.089 ERROR org.apache.spark.util.Utils: Uncaught exception in thread element-tracking-store-worker14:40:44.089 ERROR org.apache.spark.util.Utils: Uncaught exception in thread element-tracking-store-workerjava.util.NoSuchElementException: key not found: 12 at scala.collection.immutable.Map$Map1.apply(Map.scala:114) at org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$aggregateMetrics$11(SQLAppStatusListener.scala:257) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.apache.spark.sql.execution.ui.SQLAppStatusListener.aggregateMetrics(SQLAppStatusListener.scala:256) at org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$onExecutionEnd$2(SQLAppStatusListener.scala:365) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.tryLog(Utils.scala:1971) at org.apache.spark.status.ElementTrackingStore$$anon$1.run(ElementTrackingStore.scala:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)[info] - SPARK-32573: Eliminate NAAJ when BuildSide is EmptyHashedRelationWithAllNullKeys (2 seconds, 14 milliseconds)
{code}
This issue is mainly because during AQE, while sub-plan changed, the metrics update is overwrite. for example, in this UT, change from BroadcastHashJoinExec into a LocalTableScanExec, and in the onExecutionEnd action it will try aggregate all metrics during the execution, which will cause NoSuchElementException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org