[GitHub] [spark] wangyum commented on a change in pull request #24486: [SPARK-27592][SQL] Set the bucketed data source table SerDe correctly

[GitBox <gi...@apache.org>]

wangyum commented on a change in pull request #24486: [SPARK-27592][SQL] Set the bucketed data source table SerDe correctly
URL: https://github.com/apache/spark/pull/24486#discussion_r282013417
 
 

 ##########
 File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
 ##########
 @@ -358,12 +358,17 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
             "Spark SQL specific format, which is NOT compatible with Hive."
         (None, message)
 
-      // our bucketing is un-compatible with hive(different hash function)
-      case _ if table.bucketSpec.nonEmpty =>
+      // our bucketing is un-compatible with hive(different hash function).
+      // but downstream(Hive/Presto) still can read it as not bucketed table.
+      // We set the SerDe correctly and bucketing_version to spark.
+      // The downstream decides how to read it by themselves, a similar implementation:
 
 Review comment:
   Sorry. It's not bucketed table at Hive side. Related code:
   https://github.com/apache/spark/blob/f9776e389215255dc61efaa2eddd92a1fa754b48/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L444-L459
   https://github.com/apache/spark/blob/33f3c48cac087e079b9c7e342c2e58b16eaaa681/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L976-L990
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org