You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/07/31 19:52:23 UTC

(datafusion-comet) branch main updated: chore: clean up Scala compiler warnings (#5141)

This is an automated email from the ASF dual-hosted git repository.

andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/main by this push:
     new 87f7070b3a chore: clean up Scala compiler warnings (#5141)
87f7070b3a is described below

commit 87f7070b3a5af476187ff7c56550284498db556a
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Jul 31 13:52:19 2026 -0600

    chore: clean up Scala compiler warnings (#5141)
    
    * chore: clean up Scala compiler warnings
    
    Reduces Scala compiler warnings from 127 to 8 on the default Spark 4.1 /
    Scala 2.13 build, and from 15 to 1 on Spark 3.5 / Scala 2.12.
    
    The bulk of the fixes are mechanical 2.13 deprecations:
    
    - Symbol literals ('col) replaced with $"col" in TPCH.scala and
      ParquetReadSuite (63 warnings).
    - `.toIterable.asJava` replaced with `.toSeq.asJava` when handing
      collections to protobuf `addAll*` builders.
    - Implicit Array -> Seq conversions made explicit with `.toIndexedSeq`.
    - Auto-application to `()` given explicit empty argument lists
      (`SparkSession.builder()`, `Random.nextLong()`, `.count()`, etc).
    - `Long + String` replaced with string interpolation, `filterKeys` with
      `filter`, `toIterator` with `iterator`, and `hasNext()` with an
      `override def hasNext` that matches `Iterator`.
    - Two inferred existential types replaced with explicit types.
    - `Float.NaN.equals(...)` replaced with `getFloat(0).isNaN`.
    
    Two changes are slightly more than mechanical:
    
    - `GenerateDocs.categoryPages` triggered nine "adapting argument list by
      creating a 2-tuple" warnings on 2.12, because `"array" -> ("array.md",
      () => ...)` parses as a two-argument call to `->`. Introducing
      `exprPage`/`aggPage` helpers removes the ambiguity and the duplication.
      Generated docs are unchanged.
    - `FuzzDataGenerator` no longer uses the deprecated
      `RandomStringUtils.randomAlphabetic`. The replacement draws from the
      caller's seeded `Random`, so generated strings are now reproducible
      from the fuzz seed.
    
    `-processing` is added to the javac `-Xlint` list to silence two "No
    processor claimed any of these annotations" warnings.
    
    Warnings deliberately left in place, all of which need a shim or a
    larger change:
    
    - `AliasAwareOutputExpression` uses `Stream`, which is deprecated in
      2.13, but `multiTransformDown` returns `Stream` on Spark 3.x and
      `LazyList` on Spark 4.x and `LazyList` does not exist in 2.12. Note
      that this trait is unreferenced dead code and could simply be deleted.
    - `SparkThrowable.getErrorClass` was renamed to `getCondition` in Spark
      4.0.
    - `ParquetWriter.Builder.withRowGroupSize(Int)` is version-specific and
      already carries a TODO.
    - The early initializer in `CometTPCDSQuerySuite` exists so that
      `TPCDSBase`'s constructor sees the query lists.
    - `CometFuzzTestSuite` intentionally keeps `.equals` rather than the
      suggested `==`, because `==` on boxed floats uses cooperative equality
      where `NaN != NaN`. It is restructured to compare statically typed
      `AnyRef`s so the lint no longer fires.
    
    * chore: hold CI while this draft PR is under review [skip ci]
    
    * chore: move symbol literal fixes to #5168 [skip ci]
    
    The symbol literal ('col -> $"col") replacements in TPCH.scala and
    ParquetReadSuite accounted for 63 of the warnings fixed here, which is
    large enough to review on its own. They are now in #5168, so revert
    those two files back to main to avoid overlapping with that PR.
    
    * chore: drop changes now covered by #5173 and #5175
    
    The .toIterable protobuf builder fixes and the auto-application-to-()
    fixes were split out into #5173 and #5175. Remove them here so this PR
    is limited to the one-off warnings that have no category PR of their
    own. #5168 and #5170 already landed on main.
---
 pom.xml                                            |  4 +-
 .../main/scala/org/apache/comet/CometConf.scala    |  5 +-
 .../main/scala/org/apache/comet/GenerateDocs.scala | 71 ++++++++--------------
 .../apache/comet/testing/FuzzDataGenerator.scala   | 17 +++++-
 .../sql/comet/CometBroadcastExchangeExec.scala     |  2 +-
 .../shuffle/CometBlockStoreShuffleReader.scala     |  2 +-
 .../shuffle/CometNativeShuffleWriter.scala         |  2 +-
 .../org/apache/spark/sql/comet/operators.scala     |  4 +-
 .../org/apache/comet/CometFuzzTestSuite.scala      | 11 +++-
 .../comet/CometIcebergRewriteActionSuite.scala     |  9 +--
 10 files changed, 62 insertions(+), 65 deletions(-)

diff --git a/pom.xml b/pom.xml
index 318e3ca51e..2b262ae9d8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -825,7 +825,7 @@ under the License.
                   <javacArg>${java.version}</javacArg>
                   <javacArg>-target</javacArg>
                   <javacArg>${java.version}</javacArg>
-                  <javacArg>-Xlint:all,-serial,-path,-try</javacArg>
+                  <javacArg>-Xlint:all,-serial,-path,-try,-processing</javacArg>
                 </javacArgs>
                 <compilerPlugins>
                   <compilerPlugin>
@@ -922,7 +922,7 @@ under the License.
               <javacArg>${maven.compiler.source}</javacArg>
               <javacArg>-target</javacArg>
               <javacArg>${maven.compiler.target}</javacArg>
-              <javacArg>-Xlint:all,-serial,-path,-try</javacArg>
+              <javacArg>-Xlint:all,-serial,-path,-try,-processing</javacArg>
             </javacArgs>
           </configuration>
         </plugin>
diff --git a/spark/src/main/scala/org/apache/comet/CometConf.scala b/spark/src/main/scala/org/apache/comet/CometConf.scala
index bf4cced486..cdbc430829 100644
--- a/spark/src/main/scala/org/apache/comet/CometConf.scala
+++ b/spark/src/main/scala/org/apache/comet/CometConf.scala
@@ -972,7 +972,7 @@ object ConfigHelpers {
   def timeFromString(str: String, unit: TimeUnit): Long = JavaUtils.timeStringAs(str, unit)
 
   def timeToString(v: Long, unit: TimeUnit): String =
-    TimeUnit.MILLISECONDS.convert(v, unit) + "ms"
+    s"${TimeUnit.MILLISECONDS.convert(v, unit)}ms"
 
   def byteFromString(str: String, unit: ByteUnit): Long = {
     val (input, multiplier) =
@@ -984,7 +984,8 @@ object ConfigHelpers {
     multiplier * JavaUtils.byteStringAs(input, unit)
   }
 
-  def byteToString(v: Long, unit: ByteUnit): String = unit.convertTo(v, ByteUnit.BYTE) + "b"
+  def byteToString(v: Long, unit: ByteUnit): String =
+    s"${unit.convertTo(v, ByteUnit.BYTE)}b"
 }
 
 private class TypedConfigBuilder[T](
diff --git a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
index 816dbc6790..921e7b4f3e 100644
--- a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
+++ b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
@@ -25,7 +25,7 @@ import scala.collection.mutable
 import scala.collection.mutable.ListBuffer
 
 import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
-import org.apache.spark.sql.catalyst.expressions.Cast
+import org.apache.spark.sql.catalyst.expressions.{Cast, Expression}
 
 import org.apache.comet.CometConf.COMET_ONHEAP_MEMORY_OVERHEAD
 import org.apache.comet.expressions.{CometCast, CometEvalMode}
@@ -107,52 +107,29 @@ object GenerateDocs {
    * that category from the serde maps in `QueryPlanSerde`. Filenames are resolved relative to the
    * per-Spark-version compatibility/expressions directory.
    */
-  private def categoryPages: Map[String, (String, () => CategoryNotes)] = Map(
-    "array" -> ("array.md",
-    () =>
-      QueryPlanSerde.arrayExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "datetime" -> ("datetime.md",
-    () =>
-      QueryPlanSerde.temporalExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "math" -> ("math.md",
-    () =>
-      QueryPlanSerde.mathExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "struct" -> ("struct.md",
-    () =>
-      QueryPlanSerde.structExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "aggregate" -> ("aggregate.md",
-    () =>
-      QueryPlanSerde.aggrSerdeMap.toSeq.map { case (cls, serde) =>
-        aggExprNotes(cls, serde)
-      }),
-    "string" -> ("string.md",
-    () =>
-      QueryPlanSerde.stringExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "map" -> ("map.md",
-    () =>
-      QueryPlanSerde.mapExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "misc" -> ("misc.md",
-    () =>
-      QueryPlanSerde.miscExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }),
-    "url" -> ("url.md",
-    () =>
-      QueryPlanSerde.urlExpressions.toSeq.map { case (cls, serde) =>
-        exprNotes(cls, serde)
-      }))
+  private def categoryPages: Map[String, (String, () => CategoryNotes)] = {
+    // The serde maps are passed by name so that they are only read when the notes are generated.
+    def exprPage(
+        filename: String,
+        serdes: => Map[Class[_ <: Expression], CometExpressionSerde[_]])
+        : (String, () => CategoryNotes) =
+      (filename, () => serdes.toSeq.map { case (cls, serde) => exprNotes(cls, serde) })
+
+    def aggPage(filename: String, serdes: => Map[Class[_], CometAggregateExpressionSerde[_]])
+        : (String, () => CategoryNotes) =
+      (filename, () => serdes.toSeq.map { case (cls, serde) => aggExprNotes(cls, serde) })
+
+    Map(
+      "array" -> exprPage("array.md", QueryPlanSerde.arrayExpressions),
+      "datetime" -> exprPage("datetime.md", QueryPlanSerde.temporalExpressions),
+      "math" -> exprPage("math.md", QueryPlanSerde.mathExpressions),
+      "struct" -> exprPage("struct.md", QueryPlanSerde.structExpressions),
+      "aggregate" -> aggPage("aggregate.md", QueryPlanSerde.aggrSerdeMap),
+      "string" -> exprPage("string.md", QueryPlanSerde.stringExpressions),
+      "map" -> exprPage("map.md", QueryPlanSerde.mapExpressions),
+      "misc" -> exprPage("misc.md", QueryPlanSerde.miscExpressions),
+      "url" -> exprPage("url.md", QueryPlanSerde.urlExpressions))
+  }
 
   /**
    * Args:
diff --git a/spark/src/main/scala/org/apache/comet/testing/FuzzDataGenerator.scala b/spark/src/main/scala/org/apache/comet/testing/FuzzDataGenerator.scala
index 24daebe132..cf9d8a7568 100644
--- a/spark/src/main/scala/org/apache/comet/testing/FuzzDataGenerator.scala
+++ b/spark/src/main/scala/org/apache/comet/testing/FuzzDataGenerator.scala
@@ -28,12 +28,13 @@ import java.time.{Instant, LocalDateTime, ZoneId}
 import scala.collection.mutable.ListBuffer
 import scala.util.Random
 
-import org.apache.commons.lang3.RandomStringUtils
 import org.apache.spark.sql.{DataFrame, Row, SparkSession}
 import org.apache.spark.sql.types._
 
 object FuzzDataGenerator {
 
+  private val alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+
   /**
    * Date to use as base for generating temporal columns. Random integers will be added to or
    * subtracted from this value.
@@ -263,7 +264,7 @@ object FuzzDataGenerator {
             case 1 => r.nextInt().toByte.toString
             case 2 => r.nextLong().toString
             case 3 => r.nextDouble().toString
-            case 4 => RandomStringUtils.randomAlphabetic(options.maxStringLength)
+            case 4 => randomAlphabetic(r, options.maxStringLength)
             case 5 =>
               // use a constant value to trigger dictionary encoding
               "dict_encode_me!"
@@ -293,6 +294,18 @@ object FuzzDataGenerator {
     }
   }
 
+  /**
+   * Generate a random string of ASCII letters. Unlike `Random.nextString`, which draws from the
+   * whole UTF-16 range, this stays within `[A-Za-z]`.
+   */
+  private def randomAlphabetic(r: Random, length: Int): String = {
+    val sb = new StringBuilder(length)
+    while (sb.length < length) {
+      sb.append(alphabet.charAt(r.nextInt(alphabet.length)))
+    }
+    sb.toString
+  }
+
   private def randomChoice[T](list: Seq[T], r: Random): T = {
     list(r.nextInt(list.length))
   }
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala
index 9e2296067b..9ae92845c9 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala
@@ -299,7 +299,7 @@ class CometBatchRDD(
 
   override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
     val partition = split.asInstanceOf[CometBatchPartition]
-    partition.value.value.toIterator
+    partition.value.value.iterator
       .flatMap(Utils.decodeBatches(_, this.getClass.getSimpleName))
   }
 }
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometBlockStoreShuffleReader.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometBlockStoreShuffleReader.scala
index 14e656f038..3bedc0e0f4 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometBlockStoreShuffleReader.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometBlockStoreShuffleReader.scala
@@ -138,7 +138,7 @@ class CometBlockStoreShuffleReader[K, C](
 
     // Sort the output if there is a sort ordering defined.
     val resultIter = dep.keyOrdering match {
-      case Some(_: Ordering[K]) =>
+      case Some(_) =>
         throw new UnsupportedOperationException("order not allowed")
       case None =>
         aggregatedIter
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
index 7287a3599e..abbbb3f7a3 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
@@ -118,7 +118,7 @@ class CometNativeShuffleWriter[K, V](
       "output_rows" -> metricsOutputRows,
       "data_size" -> metrics("dataSize"),
       "write_time" -> metricsWriteTime) ++
-      metrics.filterKeys(detailedMetrics.contains)
+      metrics.filter { case (name, _) => detailedMetrics.contains(name) }
 
     // ShuffleWriter metrics at the root; child's metric tree underneath so the SQL UI's per-node
     // breakdown matches what the split-driver flow showed.
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala b/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala
index fcd48d69b1..343b7d41f2 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala
@@ -740,9 +740,9 @@ abstract class CometNativeExec extends CometExec {
     val (firstNonBroadcastPlanRDD, firstNonBroadcastPlanNumPartitions) =
       firstNonBroadcastPlan.get._1 match {
         case plan: CometNativeExec =>
-          (null, plan.outputPartitioning.numPartitions)
+          (null.asInstanceOf[RDD[Any]], plan.outputPartitioning.numPartitions)
         case plan =>
-          val rdd = asArrowStreamRDD(plan, 0, firstNonBroadcastSlot)
+          val rdd = asArrowStreamRDD(plan, 0, firstNonBroadcastSlot).asInstanceOf[RDD[Any]]
           (rdd, rdd.getNumPartitions)
       }
 
diff --git a/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala b/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala
index bc9738454f..1d7ddfdd5d 100644
--- a/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala
@@ -74,13 +74,13 @@ class CometFuzzTestSuite extends CometFuzzTestBase {
           case "TINYINT" | "SMALLINT" =>
             s"$defaultValueType(${defaultValueRow.get(0)})"
           case "FLOAT" =>
-            if (Float.NaN.equals(defaultValueRow.get(0))) {
+            if (defaultValueRow.getFloat(0).isNaN) {
               floatNaNLiteral
             } else {
               s"$defaultValueType(${defaultValueRow.get(0)})"
             }
           case "DOUBLE" =>
-            if (Double.NaN.equals(defaultValueRow.get(0))) {
+            if (defaultValueRow.getDouble(0).isNaN) {
               doubleNaNLiteral
             } else {
               s"$defaultValueType(${defaultValueRow.get(0)})"
@@ -109,7 +109,12 @@ class CometFuzzTestSuite extends CometFuzzTestBase {
                 .asInstanceOf[Array[Byte]]
                 .sameElements(spark.sql(sql).collect()(0).get(0).asInstanceOf[Array[Byte]]))
           } else {
-            assert(defaultValueRow.get(0).equals(spark.sql(sql).collect()(0).get(0)))
+            // Compare as boxed values with `equals` rather than `==`: for a NaN default
+            // value, boxed Float/Double equality treats NaN as equal to itself, whereas
+            // primitive `==` would not.
+            val expectedValue = defaultValueRow.get(0).asInstanceOf[AnyRef]
+            val actualValue = spark.sql(sql).collect()(0).get(0).asInstanceOf[AnyRef]
+            assert(expectedValue.equals(actualValue))
           }
         }
       }
diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala
index 53454d0034..af2508aedc 100644
--- a/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala
@@ -585,10 +585,11 @@ class CometIcebergRewriteActionSuite extends CometTestBase with CometIcebergTest
       classOrTarget: AnyRef,
       methodName: String,
       args: (Class[_], Any)*): AnyRef = {
-    val (clazz, target) = classOrTarget match {
-      case c: Class[_] => (c, null)
-      case t => (t.getClass, t)
-    }
+    // A `Class[_]` argument means a static call, so there is no receiver instance.
+    val isStatic = classOrTarget.isInstanceOf[Class[_]]
+    val clazz: Class[_] =
+      if (isStatic) classOrTarget.asInstanceOf[Class[_]] else classOrTarget.getClass
+    val target: AnyRef = if (isStatic) null else classOrTarget
     val (paramTypes, values) = args.unzip
     val method = clazz.getMethod(methodName, paramTypes: _*)
     method.setAccessible(true)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org