[jira] [Created] (SPARK-37487) CollectMetrics is executed twice if it is followed by an sort

["Tanel Kiis (Jira)" <ji...@apache.org>]

Tanel Kiis created SPARK-37487:
----------------------------------

             Summary: CollectMetrics is executed twice if it is followed by an sort
                 Key: SPARK-37487
                 URL: https://issues.apache.org/jira/browse/SPARK-37487
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.3.0
            Reporter: Tanel Kiis


It is bets examplified by this new UT in DataFrameCallbackSuite:
{code}
  test("SPARK-XXXXX: get observable metrics with sort by callback") {
    val df = spark.range(100)
      .observe(
        name = "my_event",
        min($"id").as("min_val"),
        max($"id").as("max_val"),
        // Test unresolved alias
        sum($"id"),
        count(when($"id" % 2 === 0, 1)).as("num_even"))
      .observe(
        name = "other_event",
        avg($"id").cast("int").as("avg_val"))
      .sort($"id".desc)

    validateObservedMetrics(df)
  }
{code}

The count aggregate reports twice the number of rows:
{code}
[info] - SPARK-XXXXX: get observable metrics with sort by callback *** FAILED *** (169 milliseconds)
[info]   [0,99,9900,100] did not equal [0,99,4950,50] (DataFrameCallbackSuite.scala:342)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.checkMetrics$1(DataFrameCallbackSuite.scala:342)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.validateObservedMetrics(DataFrameCallbackSuite.scala:350)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.$anonfun$new$21(DataFrameCallbackSuite.scala:324)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
{code}

I could not figure out how this happes. Hopefully the UT can help with debugging



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org