[jira] [Created] (SPARK-60386) Enforce FILE EXTERNAL in table columns and reject FILE where it would need an ordering

["Dejan Krakovic (Jira)" <ji...@apache.org>]

Dejan Krakovic created SPARK-60386:
--------------------------------------

             Summary: Enforce FILE EXTERNAL in table columns and reject FILE where it would need an ordering
                 Key: SPARK-60386
                 URL: https://issues.apache.org/jira/browse/SPARK-60386
             Project: Spark
          Issue Type: Sub-task
          Components: Spark Core
    Affects Versions: 4.4.0
            Reporter: Dejan Krakovic


Part 2 of the FileType SPIP (SPARK-59132), building on the Catalyst type added in SPARK-60092.

The SPIP requires the EXTERNAL keyword when a FILE column is declared in a table, and gives FileType no defined ordering. This ticket enforces both at analysis time.

Table columns:
 - A FILE in a table column definition must be FILE EXTERNAL, including when nested in a struct, array or map. This covers CREATE TABLE, REPLACE TABLE, ALTER TABLE ADD COLUMNS and ALTER TABLE
 REPLACE COLUMNS, for V1 and V2 tables (FILE_TYPE_MISSING_QUALIFIER).
 - CREATE TABLE AS SELECT cannot store a FILE that has no qualifier (FILE_TYPE_UNKNOWN_WRITE).
 - A FILE column cannot be a partition or clustering column (INVALID_PARTITION_COLUMN_DATA_TYPE).

 Routines:
 - Following the SPIP, functions declare a plain FILE. A FILE EXTERNAL in the parameter or return types of a SQL function is rejected (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_QUALIFIER), and
 so is a FILE in a map key of its return type (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_MAP_KEY).
 - The same applies to the return type of UDFs, both when they are registered and when they are used in a query.

 No ordering:
 - FILE is rejected as a map key (INVALID_MAP_KEY_TYPE), a grouping expression (GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE), in INTERSECT, EXCEPT and DISTINCT
 (UNSUPPORTED_FEATURE.SET_OPERATION_ON_FILE_TYPE), as a repartition expression (UNSUPPORTED_FEATURE.PARTITION_BY_FILE), and in hash functions (DATATYPE_MISMATCH.HASH_FILE_TYPE).
 - Join keys, comparisons and sorting are already rejected because FileType is not orderable (INVALID_ORDERING_TYPE). This ticket adds tests for them.

The checks for set operations and repartitioning apply to both the fixed-point analyzer and the single-pass resolver. The FILE type remains gated by spark.sql.fileType.enabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org