[jira] [Created] (SPARK-58117) Spark rewrite_data_files procedure fails to compact hash-partitioned Iceberg tables with WHERE clauses

["Hemanath (Jira)" <ji...@apache.org>]

Hemanath created SPARK-58117:
--------------------------------

             Summary: Spark rewrite_data_files procedure fails to compact hash-partitioned Iceberg tables with WHERE clauses
                 Key: SPARK-58117
                 URL: https://issues.apache.org/jira/browse/SPARK-58117
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.0.1
            Reporter: Hemanath


When attempting to run partition-by-partition compaction on Iceberg tables with hash bucket partitioning using Spark's {{rewrite_data_files}} procedure, the operation fails or produces no results. This prevents efficient compaction of large hash-partitioned tables.

*Reproducible Steps*

Create an Iceberg table with hash bucket partitioning and Attempt partition-by-partition compaction using partition field name:

pythonspark.sql("""    CALL system.rewrite_data_files(        table => 'namespace.table',        where => 'TEST_PROGRAM_bucket = 12'    )""")

 

*Getting the below error:*
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name TEST_PROGRAM_bucket cannot be resolved. Did you mean one of the following? [TEST_PROGRAM, TEST_PROGRAM_VERSION, CONTROL_HIGH, CONTROL_LOW, PRODUCT]. SQLSTATE: 42703 java.lang.IllegalArgumentException: Cannot parse predicates in where option: TEST_PROGRAM_bucket = 12



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org