You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jamesbornholt (via GitHub)" <gi...@apache.org> on 2024/11/12 20:10:51 UTC

[I] TableOperations.locationProvider is not respected by Spark [iceberg]

jamesbornholt opened a new issue, #11527:
URL: https://github.com/apache/iceberg/issues/11527

   ### Apache Iceberg version
   
   1.6.1
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   I have a custom Catalog implementation that overrides [`TableOperations.locationProvider`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableOperations.java#L86-L91). But running DML queries with Spark doesn't seem to invoke that overriden method. Instead it calls `LocationProviders.locationsFor` (the `BaseMetastoreTableOperations` implementation of `locationProvider`) directly:
   
   https://github.com/apache/iceberg/blob/11d21b26ecbb30361b2b2eee0c335d6cd9560c8d/core/src/main/java/org/apache/iceberg/SerializableTable.java#L244
   
   through this stack trace when running something like `spark.sql("INSERT INTO ...")`:
   
   ```
   org.apache.iceberg.LocationProviders.locationsFor(LocationProviders.java:42)
   org.apache.iceberg.SerializableTable.locationProvider(SerializableTable.java:244)
   org.apache.iceberg.io.OutputFileFactory$Builder.build(OutputFileFactory.java:177)
   org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:681)
   org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:668)
   org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:441)
   org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:430)
   org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:496)
   org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:393)
   ```
   
   It looks like this may have broken in #9029 (@przemekd, @aokolnychyi) -- I tried reverting that change and my `locationProvider` is now invoked as expected.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "przemekd (via GitHub)" <gi...@apache.org>.
przemekd commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-2472142955

   @jamesbornholt I see. It looks like that PR fixed one issue but also introduced new one. I was unaware that you can provide custom default location provider by implementing custom Catalog. But I still argue the spark job shouldn't fail if it just reads data from tables without providing that custom location provider implementation. 
   I can change the code to use:
   ```
   table.locationProvider()
   ```
   for getting the table's location provider, but instead of letting it possibly failing right away we could wrap that into a custom Result class such as:
   ```
   import java.io.Serializable;
   
   public class Result<T> implements Serializable {
       private static final long serialVersionUID = 1L;
   
       private final T value;
       private final Exception error;
   
       private Result(T value, Exception error) {
           this.value = value;
           this.error = error;
       }
   
       public static <T> Result<T> success(T value) {
           return new Result<>(value, null);
       }
   
       public static <T> Result<T> failure(Exception error) {
           return new Result<>(null, error);
       }
   
       public boolean isSuccess() {
           return error == null;
       }
   
       public T getValue() {
           if (error != null) {
               throw new IllegalStateException("Cannot get value from a failed result", error);
           }
           return value;
       }
   
       public Exception getError() {
           if (error == null) {
               throw new IllegalStateException("No error present in a successful result");
           }
           return error;
       }
   }
   
   ```
   and it will get only unpacked when the location provider is truly needed.
   Let me know what you think about that @jamesbornholt @aokolnychyi


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "pdames (via GitHub)" <gi...@apache.org>.
pdames commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-3303692376

   Also impacted by this bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-3330418410

   Hey @przemekd,
   
   We’re starting to see the same issue on our side. When users rely on custom catalogs with an expected location provider the SerializableTable logic ends up writing data to the wrong location.
   
   Would you be open to reopening the PR you had for this? I’d be happy to review and leave feedback. If not I can hop in!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "jamesbornholt (via GitHub)" <gi...@apache.org>.
jamesbornholt commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-2485075657

   I agree it’s desirable to allow read queries to succeed without the custom location provider available. I like the idea of just deferring the exception like this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-3303713279

   Is there a PR up for this? I see a closed one from May


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-2889302653

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "przemekd (via GitHub)" <gi...@apache.org>.
przemekd commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-3382587168

   Hey @geruh I cannot reopen my PR, it either needs to be reopened by a maintainer (?) or I can create a new one? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "przemekd (via GitHub)" <gi...@apache.org>.
przemekd commented on issue #11527:
URL: https://github.com/apache/iceberg/issues/11527#issuecomment-2920512345

   Please do not close it ;) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] TableOperations.locationProvider is not respected by Spark [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar closed issue #11527: TableOperations.locationProvider is not respected by Spark
URL: https://github.com/apache/iceberg/issues/11527


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org