[GitHub] xuanyuanking commented on a change in pull request #23277: [SPARK-26327][SQL] Metrics in FileSourceScanExec not update correctly

[GitBox <gi...@apache.org>]

xuanyuanking commented on a change in pull request #23277: [SPARK-26327][SQL] Metrics in FileSourceScanExec not update correctly
URL: https://github.com/apache/spark/pull/23277#discussion_r240517252
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
 ##########
 @@ -167,19 +167,14 @@ case class FileSourceScanExec(
       partitionSchema = relation.partitionSchema,
       relation.sparkSession.sessionState.conf)
 
+  private var fileListingTime = 0L
+
   @transient private lazy val selectedPartitions: Seq[PartitionDirectory] = {
     val optimizerMetadataTimeNs = relation.location.metadataOpsTimeNs.getOrElse(0L)
     val startTime = System.nanoTime()
     val ret = relation.location.listFiles(partitionFilters, dataFilters)
     val timeTakenMs = ((System.nanoTime() - startTime) + optimizerMetadataTimeNs) / 1000 / 1000
-
-    metrics("numFiles").add(ret.map(_.files.size.toLong).sum)
-    metrics("metadataTime").add(timeTakenMs)
-
-    val executionId = sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)
-    SQLMetrics.postDriverMetricUpdates(sparkContext, executionId,
-      metrics("numFiles") :: metrics("metadataTime") :: Nil)
-
 
 Review comment:
   All versions after 2.2.0 have this bug, this PR fix the conner case(but all case by sql) in the commit. If we test by:
   ```
   spark.read.parquet($filepath)
   ```
   the metrics show correctly cause `relation.partitionSchemaOption` is None, `selectedPartitions` initializing not triggered at wrong place. But if we test as the added UT:
   ```
   sql("select * from table partition=xx")
   ```
   the metrics goes wrong.
   I changed the JIRA description and title more detailed, please help me to check whether I explain clearly. Thanks :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org