[I] Spark: Z-order rewrite fails for TimestampNTZ columns in rewrite_data_files [iceberg]

["milleniax (via GitHub)" <gi...@apache.org>]

milleniax opened a new issue, #15777:
URL: https://github.com/apache/iceberg/issues/15777

   ## Problem
   When a Z-order column has Spark type `TimestampNTZType` (Iceberg `timestamp` without time zone),  
   `rewrite_data_files` with `strategy => 'sort'` and `sort_order => 'zorder(...)'` fails with:
   Cannot use column of type TimestampNTZType in ZOrdering, the type is unsupported
   
   ## Root Cause
   `SparkZOrderUDF#sortedLexicographically` handles `TimestampType` (cast to long for ordered bytes) but does not handle `TimestampNTZType`, so NTZ falls through to the unsupported-type branch.
   ## Steps to Reproduce
   **Environment:**
   * Iceberg: `<!-- e.g. 1.10.0 -->`
   * Spark: `<!-- e.g. 4.0.x -->`
   ```python
   from datetime import datetime
   from pyspark.sql import Row, SparkSession
   spark = # SparkSession with Iceberg catalog configured
   spark.sql("DROP TABLE IF EXISTS spark_catalog.default.check_table_ntz")
   spark.sql("""
       CREATE TABLE spark_catalog.default.check_table_ntz (
           ts timestamp_ntz,
           col_a bigint
       )
       USING iceberg
       PARTITIONED BY (days(ts))
       TBLPROPERTIES ('format-version' = '2')
   """)
   data = [
       Row(ts=datetime(2024, 1, 1, 0, 0, 0), col_a=1),
       Row(ts=datetime(2024, 1, 2, 0, 0, 0), col_a=2),
   ]
   spark.createDataFrame(data).coalesce(1).writeTo(
       "spark_catalog.default.check_table_ntz"
   ).append()
   spark.sql("""
       CALL spark_catalog.system.rewrite_data_files(
           table => 'spark_catalog.default.check_table_ntz',
           strategy => 'sort',
           sort_order => 'zorder(ts, col_a)',
           options => map('rewrite-all', 'true')
       )
   """)
   ```
   
   ## Actual Behavior
   IllegalArgumentException from the Z-order path: TimestampNTZType is reported as unsupported for Z-ordering.
   
   ## Expected Behavior
   Z-order rewrite should support timestamp_ntz columns (same ordered-bytes treatment as for TimestampType)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Spark: Z-order rewrite fails for TimestampNTZ columns in rewrite_data_files [iceberg]

["RussellSpitzer (via GitHub)" <gi...@apache.org>]

RussellSpitzer closed issue #15777: Spark: Z-order rewrite fails for TimestampNTZ columns in rewrite_data_files
URL: https://github.com/apache/iceberg/issues/15777


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org