You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/05/23 18:30:26 UTC
[I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
kevinjqliu opened a new issue, #763:
URL: https://github.com/apache/iceberg-python/issues/763
### Feature Request / Improvement
Although not in the official spec, `version-hint.txt` can be useful to read an iceberg table without a catalog.
This is useful when considering an iceberg table as a collection of files (metadata and data files) in a "directory" (s3 path). This can also be useful when ingesting iceberg tables without a catalog. An iceberg table can thus be "packaged" as a directory.
Example Usecase
* An Iceberg table is created in a service (with catalog) in the path (s3://blah/warehouse/foo/bar/)
* Reading the Iceberg table with another service by just providing the path (s3://blah/warehouse/foo/bar/)
When reading, `version-hint.txt` determines the metadata json, usually provided by querying the catalog.
When writing, `version-hint.txt` is committed with the atomic update to the catalog.
Additionally, `StaticTable` can use `version-hint.txt` to create an iceberg table from a path.
Relevant Issues:
* [apache/iceberg/#7979](https://github.com/apache/iceberg/issues/7979)
* [duckdb_iceberg/#10](https://github.com/duckdb/duckdb_iceberg/issues/10)
cc @djouallah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "srilman (via GitHub)" <gi...@apache.org>.
srilman commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2715883331
@Fokko is this issue still open for working on? For context, we had to build a PyIceberg-based Hadoop Catalog with a subset of features for backwards compatibility when moving Bodo from Iceberg-Java to PyIceberg. See https://github.com/bodo-ai/Bodo/blob/main/bodo/io/iceberg/catalog/dir.py. It would be nice to move at least the read parts to the main repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2716383319
fwiw, I just gave up and I am using duckdb to read iceberg table , pycieberg is clearly not interested in this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "RoboDonut (via GitHub)" <gi...@apache.org>.
RoboDonut commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2971370147
i see this was merged but not included in the last release, any idea when it would be included in a release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2136335963
We discussed this issue in the monthly sync and agreed that this is a useful feature. We'll first implement the read side in `pyiceberg`. The write side is complicated due to having to support multiple concurrent writers and atomic updates in blob store, such as S3.
I will raise this issue with the Java Iceberg implementation and see if there's support also to include this as part of the Iceberg spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2783472021
I submitted a small PR to allow using `version-hint.text`: https://github.com/apache/iceberg-python/pull/1887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #763: [feat] Ability to read table using `version-hint.txt`
URL: https://github.com/apache/iceberg-python/issues/763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2246746210
@lamb-russell duckdb_iceberg can read the "metadata json file" directly.
See https://github.com/steven-luabase/duckdb-iceberg-demo/issues/1#issuecomment-2215482225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2247063619
I think it is fine to add support for reading the `version-hint.txt`, but we should not produce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #763: [feat] Ability to read table using `version-hint.txt`
URL: https://github.com/apache/iceberg-python/issues/763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2716982828
@srilman Yes, I still think it would be valuable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "lamb-russell (via GitHub)" <gi...@apache.org>.
lamb-russell commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2246727745
DuckDB appears to depend on the version-hint.text file when scanning iceberg.
<img width="1230" alt="image" src="https://github.com/user-attachments/assets/9aea6acb-e909-4866-9b7f-a1826b2a79c8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2246747456
It would be great if duckdb_iceberg could support reading directly from the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2246766948
it is quite ironic, it seems the only iceberg vendor who generate hint.text is snowflake !!! go figure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [feat] Ability to read table using `version-hint.txt` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #763:
URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2971681263
The last release (0.9.1) was a patch release where we don't want to include new functionality. This will be part of 0.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org