You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/08 06:06:41 UTC
[GitHub] [iceberg] jun-he opened a new issue #3245: [Python] support iceberg base catalog in python library
jun-he opened a new issue #3245:
URL: https://github.com/apache/iceberg/issues/3245
Seems we can use ABC for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on issue #3245: [Python] support iceberg base catalog in python library
Posted by GitBox <gi...@apache.org>.
nssalian commented on issue #3245:
URL: https://github.com/apache/iceberg/issues/3245#issuecomment-938726617
Happy to work on this to help start off the other catalogs derived from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian edited a comment on issue #3245: [Python] support iceberg base catalog in python library
Posted by GitBox <gi...@apache.org>.
nssalian edited a comment on issue #3245:
URL: https://github.com/apache/iceberg/issues/3245#issuecomment-938726617
Happy to work on this to help start off the other catalogs derived from it.
@jun-he , let me know if I can take this on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on issue #3245: [Python] support iceberg base catalog in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on issue #3245:
URL: https://github.com/apache/iceberg/issues/3245#issuecomment-1031682350
Here are some additional notes/details/questions to guide this issue. I've also included a diagram of the java implementation.
- Catalog abstraction
- Operations
- List tables
- Create a table
- Drop a table
- Replace table
- Load table (returns an instance of the main table class that users will interact with)
- Rename a table
- Check that a table exists (or maybe this is done in load table. Python users will be familiar with `try catalog.load_table() except MissingTableException`)
- Table identifier (maybe not needed?)
- In python, a table identifier could just be a string and the main Table class itself could also serve as the table identifier. `Table(“foo.bar.baz”, catalog=Catalog(...))` or `Table(namespace=”foo”, database=”bar”, table=”baz”, catalog=Catalog(...))`. Loading the table from the catalog would then be `Table.load()`.
- “Loading a table” from the catalog could easily be confused with loading the actual data. Since we’ll have both of these functionalities in a single library, should we have a more explicit class name here so it’s not confused with whatever the particular in-memory table format is that’s being used, i.e. pyarrow.Table, pandas.DataFrame, numpy.ndarray, etc.? Some options:
- IcebergTable
- TableMetadata
- TableInfo
- Namespace: Does this need to be a class? Can the `catalog.listNamespaces` method just take a string as an argument?
- Should the JDBC catalog implementation be ranked higher in priority than the hive catalog?
Java implementation:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cccs-eric commented on issue #3245: [Python] support iceberg base catalog in python library
Posted by GitBox <gi...@apache.org>.
cccs-eric commented on issue #3245:
URL: https://github.com/apache/iceberg/issues/3245#issuecomment-1031695082
@samredai Is there a plan to support a filesystem-based catalog? Local filesystem, AWS S3 buckets or Azure datalake? Or that use case would be cover by `Table.load(locationUri)` assuming you know the location URI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on issue #3245: [Python] support iceberg base catalog in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on issue #3245:
URL: https://github.com/apache/iceberg/issues/3245#issuecomment-1031730083
@cccs-eric I think there are some specific challenges with a filesystem-based catalog around file locking. I'd definitely be up for doing something like that if it's possible, but we can also do something else to make it easy to keep a local catalog, such as a SQLite database with a JDBC catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org