You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "liujiayi771 (via GitHub)" <gi...@apache.org> on 2024/09/05 03:13:34 UTC

[PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

liujiayi771 opened a new pull request, #47996:
URL: https://github.com/apache/spark/pull/47996

   ### What changes were proposed in this pull request?
   Combine the options of both `DataSourceV2Relation` and `CSVTable` when constructing `CSVScanBuilder`.
   
   
   ### Why are the changes needed?
   Currently, the `CSVTable` only uses the options from relation when constructing the `CSVScanBuilder`, which leads to the omission of the contents in `CSVTable.options`. For the `TableCatalog`, the `dsOptions`` can be set into the `CSVTable.options` returned by the `TableCatalog.loadTable` method. If only the relation options are used here, the `TableCatalog` will not be able to pass `dsOptions` that contains CSV options to `CSVScan`.
   
   Combining the two options is a better option.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #47996:
URL: https://github.com/apache/spark/pull/47996#discussion_r1745631535


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala:
##########
@@ -37,8 +37,13 @@ case class CSVTable(
     userSpecifiedSchema: Option[StructType],
     fallbackFileFormat: Class[_ <: FileFormat])
   extends FileTable(sparkSession, options, paths, userSpecifiedSchema) {
-  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder =
-    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder = {
+    val finalOptions = this.options.asCaseSensitiveMap().asScala.filterNot {
+      case (k, _) => options.containsKey(k)
+    }.toMap ++ options.asScala.toMap
+    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema,
+      new CaseInsensitiveStringMap(finalOptions.asJava))

Review Comment:
   @cloud-fan orc/parquet/json/textTable do not combine the options of table and relation, while jdbcTable does merge them. I think we can add merging logic for all of them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #47996: [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder
URL: https://github.com/apache/spark/pull/47996


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "adrian-wang (via GitHub)" <gi...@apache.org>.
adrian-wang commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2342502388

   +1, LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #47996:
URL: https://github.com/apache/spark/pull/47996#discussion_r1745562413


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala:
##########
@@ -37,8 +37,13 @@ case class CSVTable(
     userSpecifiedSchema: Option[StructType],
     fallbackFileFormat: Class[_ <: FileFormat])
   extends FileTable(sparkSession, options, paths, userSpecifiedSchema) {
-  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder =
-    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder = {
+    val finalOptions = this.options.asCaseSensitiveMap().asScala.filterNot {
+      case (k, _) => options.containsKey(k)
+    }.toMap ++ options.asScala.toMap
+    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema,
+      new CaseInsensitiveStringMap(finalOptions.asJava))

Review Comment:
   do other formats have the same issue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2343881064

   To @liujiayi771 , FYI, Apache Spark 4.0.0-preview2 RC1 will be released next Monday.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #47996:
URL: https://github.com/apache/spark/pull/47996#discussion_r1745561973


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala:
##########
@@ -37,8 +37,13 @@ case class CSVTable(
     userSpecifiedSchema: Option[StructType],
     fallbackFileFormat: Class[_ <: FileFormat])
   extends FileTable(sparkSession, options, paths, userSpecifiedSchema) {
-  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder =
-    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder = {
+    val finalOptions = this.options.asCaseSensitiveMap().asScala.filterNot {
+      case (k, _) => options.containsKey(k)

Review Comment:
   do we need this? I think combining two maps will respect the keys in the later map.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2333296160

   Okay, I will add a test case, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2333072225

   Is it possible to add a test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2340447026

   @cloud-fan @dongjoon-hyun I have added a test case, could you help to review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #47996:
URL: https://github.com/apache/spark/pull/47996#issuecomment-2330893328

   Hi @cloud-fan. Could you help to review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Combine options of table and relation when constructing CSVScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #47996:
URL: https://github.com/apache/spark/pull/47996#discussion_r1745633422


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala:
##########
@@ -37,8 +37,13 @@ case class CSVTable(
     userSpecifiedSchema: Option[StructType],
     fallbackFileFormat: Class[_ <: FileFormat])
   extends FileTable(sparkSession, options, paths, userSpecifiedSchema) {
-  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder =
-    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder = {
+    val finalOptions = this.options.asCaseSensitiveMap().asScala.filterNot {
+      case (k, _) => options.containsKey(k)

Review Comment:
   I will remove the filter and combining directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder [spark]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #47996:
URL: https://github.com/apache/spark/pull/47996#discussion_r1745631535


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVTable.scala:
##########
@@ -37,8 +37,13 @@ case class CSVTable(
     userSpecifiedSchema: Option[StructType],
     fallbackFileFormat: Class[_ <: FileFormat])
   extends FileTable(sparkSession, options, paths, userSpecifiedSchema) {
-  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder =
-    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+  override def newScanBuilder(options: CaseInsensitiveStringMap): CSVScanBuilder = {
+    val finalOptions = this.options.asCaseSensitiveMap().asScala.filterNot {
+      case (k, _) => options.containsKey(k)
+    }.toMap ++ options.asScala.toMap
+    CSVScanBuilder(sparkSession, fileIndex, schema, dataSchema,
+      new CaseInsensitiveStringMap(finalOptions.asJava))

Review Comment:
   @cloud-fan orc/parquet/json/textTable do not combine the options of table and relation, while jdbcTable does merge them. I think we can add merging logic for all of them. How do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org