You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/05/14 14:45:54 UTC
[I] PyIceberg Near-Term Roadmap [iceberg-python]
kevinjqliu opened a new issue, #736:
URL: https://github.com/apache/iceberg-python/issues/736
### Feature Request / Improvement
# PyIceberg 0.7.0
The main objective of 0.7.0 is to have partitioned writes (non-exhaustive list :)
- [ ] Support partitioned appends: https://github.com/apache/iceberg-python/pull/703
- [ ] Support partial deletes: https://github.com/apache/iceberg-python/pull/569
- [x] Support for parallelizing writes: https://github.com/apache/iceberg-python/issues/428 https://github.com/apache/iceberg-python/issues/346
- [x] Support parallelized writes: https://github.com/apache/iceberg-python/pull/444
- [x] Support `table_exists` on catalog: https://github.com/apache/iceberg-python/issues/406 https://github.com/apache/iceberg-python/issues/507, fixed in https://github.com/apache/iceberg-python/pull/512
- [ ] Metadata tables: https://github.com/apache/iceberg-python/issues/511
- [ ] *Files* assigned to @Gowthami03B, PR in https://github.com/apache/iceberg-python/pull/614
- [x] *Snapshots* assigned to @Fokko in https://github.com/apache/iceberg-python/pull/524/
- [ ] *History* assigned to @ndrluis
- [ ] *Metadata log entries* @kevinjqliu (issue in https://github.com/apache/iceberg-python/issues/594): https://github.com/apache/iceberg-python/pull/667
- [ ] *Manifests* @geruh: PR in https://github.com/apache/iceberg-python/pull/717
- [ ] *Partitions* assigned to @syun64 (issue in https://github.com/apache/iceberg-python/issues/24)
- [x] *References* assigned to @geruh in https://github.com/apache/iceberg-python/pull/602
- [x] *Entries* assigned to @Fokko in https://github.com/apache/iceberg-python/pull/551
- [ ] Manifest read/write improvements:
- [ ] Implement rolling writes: https://github.com/apache/iceberg-python/issues/596 https://github.com/apache/iceberg-python/pull/650
- [ ] Caching of manifests: https://github.com/apache/iceberg-python/issues/595
# PyIceberg 0.8.0
- [ ] Table maintenance:
- [ ] Snapshot expiration
- [ ] Metadata rewrites
- [ ] Compaction
- [ ] Delete orphan files
- [ ] Catalogs:
- [ ] Snowflake catalog: https://github.com/apache/iceberg-python/pull/687
- [ ] Nessie catalog: https://github.com/apache/iceberg-python/issues/19
- [ ] BigLake catalog: https://github.com/apache/iceberg-python/issues/651
- [ ] ORC Support: https://github.com/apache/iceberg-python/issues/20
- [ ] Branch Support: https://github.com/apache/iceberg-python/issues/306
- [ ] Tag Support: https://github.com/apache/iceberg-python/issues/573
- [ ] Write with Sort Order https://github.com/apache/iceberg-python/issues/271
# PyIceberg 1.0.0
Long-term goals:
- Support Griffe to detect breaking API changes https://github.com/apache/iceberg-python/issues/334: https://github.com/apache/iceberg-python/pull/394
- Implement Arrow dataset: https://github.com/apache/iceberg-python/issues/30
- Support table maintenance operations: https://github.com/apache/iceberg-python/issues/31
- Add [View](https://iceberg.apache.org/view-spec/) support
- Add [Puffin](https://iceberg.apache.org/puffin-spec/) support
- Support engine integrations (DuckDB/Daft/Polars/Ray)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2143983698
> @Fokko can we add issues for creating tests and documentation for the new features of 0.7.0 as good first issues?
@tusharchou: Whenever you create a new feature, you need to add the unit & integration test and make the necessary changes in mkdocs as a part of that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "tusharchou (via GitHub)" <gi...@apache.org>.
tusharchou commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2125031765
@kevinjqliu @Fokko where would something like https://github.com/apache/iceberg-python/issues/402 go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2126667610
Sorry for the late reply, I was touching grass.
> @kevinjqliu @Fokko Where would something like the Iceberg Spark create_changelog_view procedure fit in this roadmap? Is that something that might be tackled as part of the other procedures under table maintenance, or is it likely to come later (1.0.0), or not at all in PyIceberg?
Thanks for bringing this up @corleyma 🙌 Some related work is being done in https://github.com/apache/iceberg-python/pull/533/ and I think PyIceberg should definitely support something like that.
> @kevinjqliu @Fokko where would something like https://github.com/apache/iceberg-python/issues/402 go?
I've added it to the overview. Once the partial deletes + partitioned writes are in, this is supported automatically. We might want to have some community discussion on the API once those two PRs land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2110652446
@kevinjqliu @Fokko Where would something like the Iceberg Spark `create_changelog_view` procedure fit in this roadmap? Is that something that might be tackled as part of the other procedures under table maintenance, or is it likely to come later (1.0.0), or not at all in PyIceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2417448465
@jaehyeon-kim That is correct. BigQuery Metastore is the replacement for BigLake Metastore. I recommend adjusting the roadmap to skip BigLake metastore and add support for BigQuery Metastore. This [PR](https://github.com/apache/iceberg/pull/11039) to the Iceberg Java libraries should be good reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "jaehyeon-kim (via GitHub)" <gi...@apache.org>.
jaehyeon-kim commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2412464577
It look BigLake metastore is going to be replaced to BigQuery metastore. Is the version 0.8.0 still relevant?
https://github.com/trinodb/trino/issues/20031#issuecomment-2310391785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "tusharchou (via GitHub)" <gi...@apache.org>.
tusharchou commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2133660186
@Fokko can we add issues for creating tests and documentation for the new features of 0.7.0 as good first issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2740576319
@kevinjqliu I took the liberty of unpinning this one in favor of https://github.com/apache/iceberg-python/issues/1818. I think we should either close this one, or clean it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "glesperance (via GitHub)" <gi...@apache.org>.
glesperance commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2755414983
@kevinjqliu whats the best palce to follow the roadmap now that we're closing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2417742241
Thanks for the context @anoopj. @jaehyeon-kim looks like #651 is a feature request. There's currently no committed date to implement it, I'll readjust the roadmap to reflect that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2755018701
Closing this one. We've made a lot of progress since this issue was opened last year.
See
* [0.7.0 release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.7.0)
* [0.8.0 release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.8.0)
* [0.9.0 release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.9.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #736: PyIceberg Near-Term Roadmap
URL: https://github.com/apache/iceberg-python/issues/736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg Near-Term Roadmap [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #736:
URL: https://github.com/apache/iceberg-python/issues/736#issuecomment-2755438192
@glesperance i just added #1856 please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org