[GitHub] Mister-Meeseeks commented on a change in pull request #23830: Decrease processing overhead on DataFrameReader CSV calls with specif…

[GitBox <gi...@apache.org>]

Mister-Meeseeks commented on a change in pull request #23830: Decrease processing overhead on DataFrameReader CSV calls with specif…
URL: https://github.com/apache/spark/pull/23830#discussion_r258121214
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
 ##########
 @@ -508,7 +508,9 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
       sparkSession.sessionState.conf.sessionLocalTimeZone)
     val filteredLines: Dataset[String] =
       CSVUtils.filterCommentAndEmpty(csvDataset, parsedOptions)
-    val maybeFirstLine: Option[String] = filteredLines.take(1).headOption
+    val maybeFirstLine: Option[String] =
+      if (userSpecifiedSchema.isEmpty)
+	filteredLines.take(1).headOption else None
 
 Review comment:
   Thanks guys for pointing that out. I pushed a new commit to the PR to address that.
   
   To go into more detail. It seems that the first line scan accomplishes three goals:
   
   - Schema inference
   - Schema enforcement when schema is specified by the user. 
   - Scanning the header string, so we know which lines to drop.
   
   If the user specifies a schema and header is false, then none of the above implies. The `linesWithoutHeader` is just a computationally wasteful noop. When header is false, `CSVHeaderChecker` takes no action. And `CSVUtils` just passes through the unfiltered iterator. 
   
   Therefore as long as schema is set, and header is false, then short-circuiting `maybeFirstLine` does not change any downstream behavior. 
   
   And for that that, we get major efficiency gains. The entire executor (or entire cluster if dynamic allocation isn't enabled) is sitting idle, waiting for 1 core to run an entire job just to get 1 line. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org