Re: [PR] [SPARK-56045][SQL] Add flag for ignoring Parquet UNKNOWN type annotation and revert to old behavior [spark]

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun commented on code in PR #54870:
URL: https://github.com/apache/spark/pull/54870#discussion_r2954751180


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1670,6 +1670,17 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val PARQUET_READER_RESPECT_UNKNOWN_TYPE_ANNOTATION =
+    buildConf("spark.sql.parquet.reader.respectUnknownTypeAnnotation.enabled")
+      .internal()
+      .doc("When enabled, respects the UNKNOWN type annotation in Parquet files during schema " +
+        "inference and infers NullType. When disabled, ignores the UNKNOWN annotation " +
+        "and uses the physical type instead.")
+      .version("4.1.2")
+      .withBindingPolicy(ConfigBindingPolicy.SESSION)

Review Comment:
   Hi, @ZiyaZa and @cloud-fan .
   
   This broken `branch-4.1`. Let me revert this from `branch-4.1` only for now.
   
   ```
   [error] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:1627:8: value withBindingPolicy is not a member of org.apache.spark.internal.config.ConfigBuilder
   [error] possible cause: maybe a semicolon is missing before `value withBindingPolicy`?
   [error]       .withBindingPolicy(ConfigBindingPolicy.SESSION)
   [error]        ^
   [error] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:1627:26: not found: value ConfigBindingPolicy
   [error]       .withBindingPolicy(ConfigBindingPolicy.SESSION)
   [error]                          ^
   [error] two errors found
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org