[GitHub] [spark] grundprinzip commented on a diff in pull request #42099: [SPARK-44505][SQL] Provide override for columnar support in Scan for DSv2

["grundprinzip (via GitHub)" <gi...@apache.org>]

grundprinzip commented on code in PR #42099:
URL: https://github.com/apache/spark/pull/42099#discussion_r1270303330


##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:
##########
@@ -125,4 +125,26 @@ default CustomMetric[] supportedCustomMetrics() {
   default CustomTaskMetric[] reportDriverMetrics() {
     return new CustomTaskMetric[]{};
   }
+
+  /**
+   * This enum defines how the columnar support for the partitions of the data source
+   * should be determined. The default value is `PARTITION_DEFINED` which indicates that each
+   * partition can deterine if it should be columnar or not. SUPPORTED and UNSUPPORTED provide
+   * default shortcuts to indicate support for columnar data or not.
+   *
+   * @since 3.5.0
+   */
+  enum ColumnarSupportType {
+    PARTITION_DEFINED,
+    SUPPORTED,
+    UNSUPPORTED
+  }
+
+  /**
+   * Subclasses can implement this method to indicate if the support for columnar data should
+   * be determined by each partition or is set as a default for the whole scan.
+   *
+   * @since 3.5.0
+   */
+  default ColumnarSupportType supportsColumnar() { return ColumnarSupportType.PARTITION_DEFINED; }

Review Comment:
   it can't be a boolean because there are three different values. 
   
   Since this is used as an override, we must preserve today's behavior for backwards compatibility. The backward compatible behavior is to iterate over all partitions to check. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org