You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tanel Kiis (Jira)" <ji...@apache.org> on 2021/05/03 15:59:00 UTC
[jira] [Comment Edited] (SPARK-35296) Dataset.observe fails with an
assertion
[ https://issues.apache.org/jira/browse/SPARK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338438#comment-17338438 ]
Tanel Kiis edited comment on SPARK-35296 at 5/3/21, 3:58 PM:
-------------------------------------------------------------
I tried to change an excisting UT to reproduce this:
{code}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
index b3d29df1b2..0fcd31a09c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
@@ -246,7 +246,7 @@ class DataFrameCallbackSuite extends QueryTest
}
spark.listenerManager.register(listener)
try {
- val df = spark.range(100)
+ val df = spark.range(100).repartition(2)
.observe(
name = "my_event",
min($"id").as("min_val"),
{code}
But did not hit the same exception:
{code}
[info] - get observable metrics by callback *** FAILED *** (324 milliseconds)
[info] 0 did not equal 2 (DataFrameCallbackSuite.scala:261)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info] at org.apache.spark.sql.util.DataFrameCallbackSuite.checkMetrics$1(DataFrameCallbackSuite.scala:261)
[info] at org.apache.spark.sql.util.DataFrameCallbackSuite.$anonfun$new$15(DataFrameCallbackSuite.scala:270)
{code}
Although i think that this should not happen either.
was (Author: tanelk):
I tried to change an excisting UT to reproduce this:
{code}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
index b3d29df1b2..0fcd31a09c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
@@ -246,7 +246,7 @@ class DataFrameCallbackSuite extends QueryTest
}
spark.listenerManager.register(listener)
try {
- val df = spark.range(100)
+ val df = spark.range(100).repartition(2)
.observe(
name = "my_event",
min($"id").as("min_val"),
{code}
But did not hit the same exception:
{code}
[info] - get observable metrics by callback *** FAILED *** (324 milliseconds)
[info] 0 did not equal 2 (DataFrameCallbackSuite.scala:261)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info] at org.apache.spark.sql.util.DataFrameCallbackSuite.checkMetrics$1(DataFrameCallbackSuite.scala:261)
[info] at org.apache.spark.sql.util.DataFrameCallbackSuite.$anonfun$new$15(DataFrameCallbackSuite.scala:270)
{code}
> Dataset.observe fails with an assertion
> ---------------------------------------
>
> Key: SPARK-35296
> URL: https://issues.apache.org/jira/browse/SPARK-35296
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Tanel Kiis
> Priority: Major
>
> I hit this assertion error when using dataset.observe:
> {code}
> java.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:208) ~[scala-library-2.12.10.jar:?]
> at org.apache.spark.sql.execution.AggregatingAccumulator.setState(AggregatingAccumulator.scala:204) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.execution.CollectMetricsExec.$anonfun$doExecute$2(CollectMetricsExec.scala:72) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.execution.CollectMetricsExec.$anonfun$doExecute$2$adapted(CollectMetricsExec.scala:71) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:125) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContextImpl.$anonfun$markTaskCompleted$1(TaskContextImpl.scala:124) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContextImpl.$anonfun$markTaskCompleted$1$adapted(TaskContextImpl.scala:124) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContextImpl.$anonfun$invokeListeners$1(TaskContextImpl.scala:137) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContextImpl.$anonfun$invokeListeners$1$adapted(TaskContextImpl.scala:135) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.10.jar:?]
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.10.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.10.jar:?]
> at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:135) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:124) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.scheduler.Task.run(Task.scala:147) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) ~[spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) [spark-core_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) [spark-core_2.12-3.1.1.jar:3.1.1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
> {code}
> A workaround, that I used was to add .coalesce(1) before calling this method.
> It happens in a quite complex query and I have not been able to reproduce this with a simpler query
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org