You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "haizhou-zhao (via GitHub)" <gi...@apache.org> on 2025/06/02 06:15:53 UTC

[I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

haizhou-zhao opened a new issue, #11554:
URL: https://github.com/apache/iceberg/issues/11554

   ### Apache Iceberg version
   
   None
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   Part of: https://github.com/apache/iceberg/issues/11079
   
   ## Intro
   The test `TestCreateTable::testCreateTableCommitProperties` will only work on Hadoop/Hive Catalog but not on REST Catalog. Code link: https://github.com/apache/iceberg/blame/0a705b0637db484730eb4eece69ae6c4d52fd9da/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java#L353
   
   ## Error message
   ```
   TestCreateTable > testCreateTableCommitProperties() > catalogName = testrest, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=rest, cache-enabled=false, uri=http://localhost:34885/} FAILED
       java.lang.AssertionError: 
       Expecting actual throwable to be an instance of:
         org.apache.iceberg.exceptions.ValidationException
       but was:
         org.apache.iceberg.exceptions.BadRequestException: Malformed request: Table property commit.retry.num-retries must have integer value
       	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:208)
       	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:118)
       	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:102)
       	...(164 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)
           at org.apache.iceberg.spark.sql.TestCreateTable.testCreateTableCommitProperties(TestCreateTable.java:366)
   ```
   
   ## Root cause
   REST client will throw `BadRequestException` (`HTTP 400`) where a `ValidationException` will be thrown by Hive/Hadoop Catalog in similar situation. REST client will simply categorize `ValidationException` as `HTTP 400` and throw `BadRequestException` instead. This is currently determined by REST's `ErrorHandlers`: https://github.com/apache/iceberg/blob/0a705b0637db484730eb4eece69ae6c4d52fd9da/core/src/main/java/org/apache/iceberg/rest/ErrorHandlers.java#L202
   
   ## Further consideration
   Two options going forward:
   1. Unify Hive/Hadoop/REST interface so all of them throw the same type of exception in the cases that this test is covering for
   2. Accept that Hive/Hadoop/REST could throw different type of exception in these cases and modify the test itself
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "nastra (via GitHub) via issues" <is...@iceberg.apache.org>.
nastra closed issue #11554: [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog
URL: https://github.com/apache/iceberg/issues/11554


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "haizhou-zhao (via GitHub)" <gi...@apache.org>.
haizhou-zhao commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-2477732940

   cc: @dramaticlly (author of the test in the issue)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "rshdhere (via GitHub)" <gi...@apache.org>.
rshdhere commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-6129629193

   Hi @haizhou-zhao, thanks for writing this up so clearly, and thanks @dramaticlly for the pointer to #9225.
   
   I checked current `main` and this still applies. `testCreateTableCommitProperties` is still skipped for the REST catalog in all four Spark versions ([v4.2 for example](https://github.com/apache/iceberg/blob/256f542d3e9f50f97fabf4ffea498f151fb44820/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java#L344-L347)). `DefaultErrorHandler` turns a 400 back into an `IllegalArgumentException` when the error type matches, but it has nothing similar for `ValidationException` ([here](https://github.com/apache/iceberg/blob/256f542d3e9f50f97fabf4ffea498f151fb44820/core/src/main/java/org/apache/iceberg/rest/ErrorHandlers.java#L337-L341)).
   
   I'd like to pick this up if that's okay. Building on @dramaticlly's idea, I'd add the same kind of mapping for `ValidationException` in `ErrorHandlers` with a unit test, then drop the `assumeThat` so the test runs against REST too. If you'd rather keep it as a `BadRequestException` and only adjust the test, I'm happy to go that way instead.
   
   Could someone assign this to me if either direction sounds good? Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "rshdhere (via GitHub)" <gi...@apache.org>.
rshdhere commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-6132265770

   Thanks @dramaticlly, that's good to know! I've opened #18473 with the fix. It follows the same approach as #9225: `DefaultErrorHandler` now rethrows a `ValidationException` when a 400 comes back with that type, and `testCreateTableCommitProperties` now runs against the REST catalog in all four Spark versions. Would appreciate a look whenever you have time.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-2917892025

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-6130897941

   @rshdhere in general iceberg issue is not assigned, but feel free to share the PR with the proposed fix and link to this issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #11554:
URL: https://github.com/apache/iceberg/issues/11554#issuecomment-2480067675

   thank you @haizhou-zhao , I realized we actually have similar special handling for IllegalArgumentException per #9225 and I am wondering if it's worth doing the same for ValidationException. 
   
   The original reason I use VE instead of IAE is because it's already being used in 
      https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableMetadata.java#L135
   and 
   https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetricsConfig.java#L204-L213
   
   I think 400 generally is correct behaviour as user need to change their input, as of now the easier thing is probably add `assumeThat` so that this particular test will only run against the hive/hadoop catalog


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11554: [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog
URL: https://github.com/apache/iceberg/issues/11554


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org