[GitHub] [iceberg] aokolnychyi edited a comment on issue #1306: Spark 3: Consider providing better support for path-based tables

[GitBox <gi...@apache.org>]

aokolnychyi edited a comment on issue #1306:
URL: https://github.com/apache/iceberg/issues/1306#issuecomment-671588572


   > CREATE TABLE "name" USING iceberg OPTIONS (path = "x")
   
   @RussellSpitzer, create table statements are delegated to either `HadoopCatalog` or `HiveCatalog` in Iceberg. The example above can be processed only by `HiveCatalog` as we cannot set a table location in `HadoopCatalog`. While `HiveCatalog` allows arbitrary table locations, it requires HMS. While `HadoopCatalog` doesn't need HMS, it requires a special layout and does list operations. We have one more API to work with Iceberg tables: `HadoopTables`. As opposed to `HadoopCatalog`, the location can be arbitrary.
   
   I see a lot of cases when people just refer to a table by location. While we can load such tables, we cannot create them through Spark and there is no SQL support. I wonder if there is something we could do differently and provide better user experience with tables identified by path.
   
   > I think there may already be support for tables that are identified using ```iceberg.`fs:/path/to/table````. 
   
   @rdblue, I actually forgot about the SQL on files syntax. Unfortunately, it works only with built-in v1 internal file sources. Ideally, we should see whether we can enable support for such syntax in all V2 data sources. Even if we do so, this will only address query support. Users will have to call Iceberg APIs to create tables. An alternative is to interpret path identifiers in our Spark catalogs. For example, we could support ```iceberg_hadoop_catalog.`path/to/table/` ``` syntax.
   
   > Isn't the main blocker that we need an atomic operation? HadoopTableOperations works for HDFS tables, but any other file system, including file:, needs something for coordination. I'm not sure what that would be for path-based tables.
   
   I'd just use `HadoopTables` and document the limitations.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org