You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "lsm1 (via GitHub)" <gi...@apache.org> on 2026/02/03 05:20:03 UTC

[PR] Optimize native metrics retrieval by passing keys directly [auron]

lsm1 opened a new pull request, #1982:
URL: https://github.com/apache/auron/pull/1982

   # Which issue does this PR close?
   None
   
   # Rationale for this change
    The previous native metrics retrieval fetched all default metrics for every operator. This changes allows specific operators to request only the relevant metric keys, reducing unnecessary overhead.
   
   # What changes are included in this PR?
    Updated `getDefaultNativeMetrics` to accept a `Set[String]` of metric keys instead of returning a fixed map.
   
   # Are there any user-facing changes?
   No
   
   # How was this patch tested?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #1985] Optimize native metrics retrieval by passing keys directly [auron]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1982:
URL: https://github.com/apache/auron/pull/1982#discussion_r3032037467


##########
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeWindowBase.scala:
##########
@@ -67,8 +67,7 @@ abstract class NativeWindowBase(
 
   override lazy val metrics: Map[String, SQLMetric] = SortedMap[String, SQLMetric]() ++ Map(
     NativeHelper
-      .getDefaultNativeMetrics(sparkContext)
-      .filterKeys(Set("stage_id", "output_rows", "elapsed_compute"))
+      .getDefaultNativeMetrics(sparkContext, Set("stage_id", "output_rows", "elapsed_compute"))
       .toSeq: _*)

Review Comment:
   `getDefaultNativeMetrics` now returns a `Map` directly, but this call site still does `.toSeq` and rebuilds another `Map` just to merge into a `SortedMap`. You can reduce allocations (and keep deterministic ordering) by merging the returned map directly (e.g., `SortedMap.empty ++ NativeHelper.getDefaultNativeMetrics(...)`).



##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala:
##########
@@ -233,7 +245,8 @@ object NativeHelper extends Logging {
   }
 
   def getNativeFileScanMetrics(sc: SparkContext): Map[String, SQLMetric] = TreeMap(
-    getDefaultNativeMetrics(sc)
-      .filterKeys(Set("stage_id", "output_rows", "elapsed_compute"))
-      .toSeq ++ getDefaultNativeFileMetrics(sc).toSeq: _*)
+    getDefaultNativeMetrics(
+      sc,
+      Set("stage_id", "output_rows", "elapsed_compute")).toSeq ++ getDefaultNativeFileMetrics(
+      sc).toSeq: _*)

Review Comment:
   `getNativeFileScanMetrics` still materializes `getDefaultNativeMetrics(...).toSeq` and reconstructs a `TreeMap` via varargs. Since `getDefaultNativeMetrics` now returns a map directly, you can avoid the intermediate sequence/varargs by combining maps with `++` (e.g., `getDefaultNativeMetrics(...) ++ getDefaultNativeFileMetrics(sc)`), preserving sorted output while being a bit lighter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #1985] Optimize native metrics retrieval by passing keys directly [auron]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas merged PR #1982:
URL: https://github.com/apache/auron/pull/1982


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #1985] Optimize native metrics retrieval by passing keys directly [auron]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1982:
URL: https://github.com/apache/auron/pull/1982#discussion_r2762558400


##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala:
##########
@@ -167,38 +167,50 @@ object NativeHelper extends Logging {
       })
   }
 
-  def getDefaultNativeMetrics(sc: SparkContext): Map[String, SQLMetric] = {
-    def metric(name: String) = SQLMetrics.createMetric(sc, name)
-    def nanoTimingMetric(name: String) = SQLMetrics.createNanoTimingMetric(sc, name)
-    def sizeMetric(name: String) = SQLMetrics.createSizeMetric(sc, name)
-
-    var metrics = TreeMap(
-      "stage_id" -> metric("stageId"),
-      "output_rows" -> metric("Native.output_rows"),
-      "output_batches" -> metric("Native.output_batches"),
-      "elapsed_compute" -> nanoTimingMetric("Native.elapsed_compute"),
-      "build_hash_map_time" -> nanoTimingMetric("Native.build_hash_map_time"),
-      "probed_side_hash_time" -> nanoTimingMetric("Native.probed_side_hash_time"),
-      "probed_side_search_time" -> nanoTimingMetric("Native.probed_side_search_time"),
-      "probed_side_compare_time" -> nanoTimingMetric("Native.probed_side_compare_time"),
-      "build_output_time" -> nanoTimingMetric("Native.build_output_time"),
-      "fallback_sort_merge_join_time" -> nanoTimingMetric("Native.fallback_sort_merge_join_time"),
-      "mem_spill_count" -> metric("Native.mem_spill_count"),
-      "mem_spill_size" -> sizeMetric("Native.mem_spill_size"),
-      "mem_spill_iotime" -> nanoTimingMetric("Native.mem_spill_iotime"),
-      "disk_spill_size" -> sizeMetric("Native.disk_spill_size"),
-      "disk_spill_iotime" -> nanoTimingMetric("Native.disk_spill_iotime"),
-      "shuffle_write_total_time" -> nanoTimingMetric("Native.shuffle_write_total_time"),
-      "shuffle_read_total_time" -> nanoTimingMetric("Native.shuffle_read_total_time"))
-
-    if (AuronAdaptor.getInstance.getAuronConfiguration.getBoolean(
-        SparkAuronConfiguration.INPUT_BATCH_STATISTICS_ENABLE)) {
-      metrics ++= TreeMap(
-        "input_batch_count" -> metric("Native.input_batches"),
-        "input_row_count" -> metric("Native.input_rows"),
-        "input_batch_mem_size" -> sizeMetric("Native.input_mem_bytes"))
+  private val defaultNativeMetricCreators: Map[String, SparkContext => SQLMetric] = Map(
+    "stage_id" -> (sc => SQLMetrics.createMetric(sc, "stageId")),
+    "output_rows" -> (sc => SQLMetrics.createMetric(sc, "Native.output_rows")),
+    "output_batches" -> (sc => SQLMetrics.createMetric(sc, "Native.output_batches")),
+    "elapsed_compute" -> (sc => SQLMetrics.createNanoTimingMetric(sc, "Native.elapsed_compute")),
+    "build_hash_map_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.build_hash_map_time")),
+    "probed_side_hash_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_hash_time")),
+    "probed_side_search_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_search_time")),
+    "probed_side_compare_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_compare_time")),
+    "build_output_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.build_output_time")),
+    "fallback_sort_merge_join_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.fallback_sort_merge_join_time")),
+    "mem_spill_count" -> (sc => SQLMetrics.createMetric(sc, "Native.mem_spill_count")),
+    "mem_spill_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.mem_spill_size")),
+    "mem_spill_iotime" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.mem_spill_iotime")),
+    "disk_spill_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.disk_spill_size")),
+    "disk_spill_iotime" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.disk_spill_iotime")),
+    "shuffle_write_total_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.shuffle_write_total_time")),
+    "shuffle_read_total_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.shuffle_read_total_time")),
+    "input_batch_count" -> (sc => SQLMetrics.createMetric(sc, "Native.input_batches")),
+    "input_row_count" -> (sc => SQLMetrics.createMetric(sc, "Native.input_rows")),
+    "input_batch_mem_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.input_mem_bytes")))
+
+  def getDefaultNativeMetrics(sc: SparkContext, keys: Set[String]): Map[String, SQLMetric] = {
+    val enabledKeys =
+      if (AuronAdaptor.getInstance.getAuronConfiguration.getBoolean(
+          SparkAuronConfiguration.INPUT_BATCH_STATISTICS_ENABLE)) {
+        keys
+      } else {
+        keys -- Set("input_batch_count", "input_row_count", "input_batch_mem_size")
+      }
+
+    TreeMap[String, SQLMetric]() ++ enabledKeys.flatMap { key =>

Review Comment:
   `enabledKeys` is a `Set`, so `enabledKeys.flatMap { ... }` produces a `Set[(String, SQLMetric)]`. That forces hashing/equality on `SQLMetric` instances and allocates an intermediate Set before building the TreeMap. To keep the optimization goal, consider iterating (`enabledKeys.iterator.flatMap(...)`) and accumulating directly into the TreeMap (or using `enabledKeys.iterator.map(...)` + `collect`), which avoids building a Set of metrics first.
   ```suggestion
       TreeMap[String, SQLMetric]() ++ enabledKeys.iterator.flatMap { key =>
   ```



##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala:
##########
@@ -167,38 +167,50 @@ object NativeHelper extends Logging {
       })
   }
 
-  def getDefaultNativeMetrics(sc: SparkContext): Map[String, SQLMetric] = {
-    def metric(name: String) = SQLMetrics.createMetric(sc, name)
-    def nanoTimingMetric(name: String) = SQLMetrics.createNanoTimingMetric(sc, name)
-    def sizeMetric(name: String) = SQLMetrics.createSizeMetric(sc, name)
-
-    var metrics = TreeMap(
-      "stage_id" -> metric("stageId"),
-      "output_rows" -> metric("Native.output_rows"),
-      "output_batches" -> metric("Native.output_batches"),
-      "elapsed_compute" -> nanoTimingMetric("Native.elapsed_compute"),
-      "build_hash_map_time" -> nanoTimingMetric("Native.build_hash_map_time"),
-      "probed_side_hash_time" -> nanoTimingMetric("Native.probed_side_hash_time"),
-      "probed_side_search_time" -> nanoTimingMetric("Native.probed_side_search_time"),
-      "probed_side_compare_time" -> nanoTimingMetric("Native.probed_side_compare_time"),
-      "build_output_time" -> nanoTimingMetric("Native.build_output_time"),
-      "fallback_sort_merge_join_time" -> nanoTimingMetric("Native.fallback_sort_merge_join_time"),
-      "mem_spill_count" -> metric("Native.mem_spill_count"),
-      "mem_spill_size" -> sizeMetric("Native.mem_spill_size"),
-      "mem_spill_iotime" -> nanoTimingMetric("Native.mem_spill_iotime"),
-      "disk_spill_size" -> sizeMetric("Native.disk_spill_size"),
-      "disk_spill_iotime" -> nanoTimingMetric("Native.disk_spill_iotime"),
-      "shuffle_write_total_time" -> nanoTimingMetric("Native.shuffle_write_total_time"),
-      "shuffle_read_total_time" -> nanoTimingMetric("Native.shuffle_read_total_time"))
-
-    if (AuronAdaptor.getInstance.getAuronConfiguration.getBoolean(
-        SparkAuronConfiguration.INPUT_BATCH_STATISTICS_ENABLE)) {
-      metrics ++= TreeMap(
-        "input_batch_count" -> metric("Native.input_batches"),
-        "input_row_count" -> metric("Native.input_rows"),
-        "input_batch_mem_size" -> sizeMetric("Native.input_mem_bytes"))
+  private val defaultNativeMetricCreators: Map[String, SparkContext => SQLMetric] = Map(
+    "stage_id" -> (sc => SQLMetrics.createMetric(sc, "stageId")),
+    "output_rows" -> (sc => SQLMetrics.createMetric(sc, "Native.output_rows")),
+    "output_batches" -> (sc => SQLMetrics.createMetric(sc, "Native.output_batches")),
+    "elapsed_compute" -> (sc => SQLMetrics.createNanoTimingMetric(sc, "Native.elapsed_compute")),
+    "build_hash_map_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.build_hash_map_time")),
+    "probed_side_hash_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_hash_time")),
+    "probed_side_search_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_search_time")),
+    "probed_side_compare_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.probed_side_compare_time")),
+    "build_output_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.build_output_time")),
+    "fallback_sort_merge_join_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.fallback_sort_merge_join_time")),
+    "mem_spill_count" -> (sc => SQLMetrics.createMetric(sc, "Native.mem_spill_count")),
+    "mem_spill_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.mem_spill_size")),
+    "mem_spill_iotime" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.mem_spill_iotime")),
+    "disk_spill_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.disk_spill_size")),
+    "disk_spill_iotime" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.disk_spill_iotime")),
+    "shuffle_write_total_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.shuffle_write_total_time")),
+    "shuffle_read_total_time" -> (sc =>
+      SQLMetrics.createNanoTimingMetric(sc, "Native.shuffle_read_total_time")),
+    "input_batch_count" -> (sc => SQLMetrics.createMetric(sc, "Native.input_batches")),
+    "input_row_count" -> (sc => SQLMetrics.createMetric(sc, "Native.input_rows")),
+    "input_batch_mem_size" -> (sc => SQLMetrics.createSizeMetric(sc, "Native.input_mem_bytes")))
+

Review Comment:
   Changing `getDefaultNativeMetrics` from a 1-arg method to a 2-arg method is a source/binary breaking change for any downstream code compiled against this module. If this object is part of a published API surface, consider keeping an overloaded `getDefaultNativeMetrics(sc: SparkContext)` (possibly deprecated) that delegates to the new implementation with the full default key set.
   ```suggestion
   
     def getDefaultNativeMetrics(sc: SparkContext): Map[String, SQLMetric] = {
       getDefaultNativeMetrics(sc, defaultNativeMetricCreators.keySet)
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org