You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "YanivZalach (via GitHub)" <gi...@apache.org> on 2026/03/20 21:38:22 UTC

[PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

YanivZalach opened a new pull request, #15706:
URL: https://github.com/apache/iceberg/pull/15706

   Tables with a column named `ICEZVALUE` fail with a misleading error during
   Z-order rewrite because `SparkZOrderFileRewriteRunner` internally uses a
   column with that name.
   
   Adds an early `Preconditions.checkArgument` in `SparkZOrderFileRewriteRunner`
   before any DataFrame operation, throwing a clear exception
   if the table schema already contains a column named `ICEZVALUE`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4103249119

   I explain why not just auto fixing it or choosing a unique name for the column isn't better.
   
   
   Express your opinion in a haiku


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on code in PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#discussion_r2968573098


##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -1766,6 +1766,16 @@ public void testCommitStateUnknownException() {
     shouldHaveSnapshots(table, 2); // Commit actually Succeeded
   }
 
+  @TestTemplate
+  public void testZOrderWithZColumnCollision() {
+    Table table = createTable(1);
+    table.updateSchema().addColumn("ICEZVALUE", Types.StringType.get()).commit();

Review Comment:
   You need to call `table.refresh()` before calling `table.updateSchema()`.
   In other usages of `createTable(int)` in this file, either `table.refresh()` is called directly or some other method is called (e.g., `shouldHaveFiles(Table, int)`) that calls it.
   This is the reason for CI failing:
   ```
       org.apache.iceberg.exceptions.CommitFailedException: Version 3 already exists: file:/tmp/junit-15785874094065309995/metadata/v3.metadata.json
           at app//org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:370)
           at app//org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:162)
           at app//org.apache.iceberg.SchemaUpdate.commit(SchemaUpdate.java:474)
           at app//org.apache.iceberg.spark.actions.TestRewriteDataFilesAction.testZOrderWithZColumnCollision(TestRewriteDataFilesAction.java:1772)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4103615205

   More seriously, I think the clearer message is probably fine , I think in reality there is almost zero chance this code path is ever executed, but if allow any schema is set and the user defined icezvalue is optional, this could silently erase that value. I doubt this will ever happen but I'll accept a precondition and test. It shouldn't add that much burden


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "YanivZalach (via GitHub)" <gi...@apache.org>.
YanivZalach commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4102942233

   I agree that option 2 (fail with a clear error) is the right tradeoff here.
   The name `ICEZVALUE` is unlikely to conflict in practice, but when it does,
   the current behavior is silent data modification, followed by a completely
   misleading error that gives the user no indication of what went wrong or how
   to fix it, and when a rewrite did not happen because of the thresholds, no error at all.
   
   The value of this PR is not preventing a common case — it's making a rare but
   confusing failure mode diagnosable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #15706:
URL: https://github.com/apache/iceberg/pull/15706


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4101879529

   > I think that it would be better to use a configurable column name than a hardcoded "ICEZVALUE" in `SparkZOrderFileRewriteRunner`.
   > 
   > We could add an option to https://iceberg.apache.org/docs/latest/spark-procedures/#options-for-sort-strategy-with-zorder-sort_order, for example, for configuring this column name. What do you think, @RussellSpitzer?
   > 
   > We should still throw the error as proposed here if the column coincides with an existing column in the table.
   
   I really don't think we need customization here, is this really that likely to be conflicting?  
   
   If we are really worried I would just randomly change the name if the column exists. My hesitation on all of this is that it doesn't seem like a very likely situation so while it is true it could happen is it worth adding code for the very rare chance it does?
   
   So in terms of solutions I would consider
   1. Randomly assign the column name (gen_zorder_xx) where if for some reason that exists we randomize the xx to something else
   2. Just fail if the chosen column name is already in use
   3. Let users customize it
   
   The order here is
   1. No user intervention needed, we hide the problem
   2. We expose that it's a problem
   3. We expose it's a problem and have to support a custom new parameter just for this special case
   
   For all of these we still have the question of, "is it really worth adding more code to the project to avoid this"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "sfc-gh-rspitzer (via GitHub)" <gi...@apache.org>.
sfc-gh-rspitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4135905057

   @wypoon are you +1 on this PR? Want to make sure we are in consensus before merging. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4101735531

   I think that it would be better to use a configurable column name than a hardcoded "ICEZVALUE" in `SparkZOrderFileRewriteRunner`.
   We could add an option to https://iceberg.apache.org/docs/latest/spark-procedures/#options-for-sort-strategy-with-zorder-sort_order, for example, for configuring this column name. What do you think, @RussellSpitzer?
   We should still throw the error as proposed here if the column coincides with an existing column in the table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "YanivZalach (via GitHub)" <gi...@apache.org>.
YanivZalach commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4103625118

   I think there might be a misunderstanding, my focus is entirely on predictability.
   
   As shown in the code example I provided in the issue, there is currently a case where the error is not thrown at all…


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4128929221

   I think I would just keep it as is, if it ever comes up we can do something more complicated later. I honestly just don't think it will ever come up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "YanivZalach (via GitHub)" <gi...@apache.org>.
YanivZalach commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4103592562

   I agree with you.
   
   The haiku:
   
   Fixed names shall stay true,
   
   Clear errors guide the user,
   
   Keep the system clean


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4103611766

   Iambic Pentameter, would also be clearer for end users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4101749642

   For ease of review, it is better to make the changes only to the latest supported version of Spark, and then backport the changes to earlier versions once the PR is merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Validate Z-order rewrite does not conflict with internal ICEZVALUE column name [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #15706:
URL: https://github.com/apache/iceberg/pull/15706#issuecomment-4112497886

   I agree that making the column name be configurable is overkill, that the complexity is not worth it. I think that 1 would be best (no user intervention needed, we hide the problem). In fact, generating a random name was my first thought.
   I agree that a table having "ICEZVALUE" as a column name is unlikely. @YanivZalach did you discover the problem because you were perusing the code and came upon this hardcoded name, or did you have a table with that column name and ran into the problem?
   Unlikely as "ICEZVALUE" may be as a column name, I'd suggest as an improvement, to use a name as @RussellSpitzer suggested, gen_zorder_xx with a randomized xx, which will almost certainly never be a column in an actual table. Then just do the precondition check, but with no need to change the randomized name if there is a match.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org