You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2024/01/26 19:55:53 UTC
[I] Support writing to a branch [iceberg-python]
Fokko opened a new issue, #306:
URL: https://github.com/apache/iceberg-python/issues/306
### Feature Request / Improvement
Right now we hardcoded that we write to the main branch all the time, would be great to make this configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913305829
We first need a `create branch` API.
Then update places currently gated by `MAIN_BRANCH`.
1. https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/pyiceberg/table/metadata.py#L124
2. https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/pyiceberg/table/__init__.py#L592
And finally add a test for writing to a branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2113260320
@Gowthami03B definitely, I've assigned the issue to you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913303260
In order to write to a branch, the branch needs to be created first.
From https://iceberg.apache.org/docs/latest/spark-writes/#writing-to-branches:
> the branch must exist before performing the write. The operation does not create the branch if it does not exist.
The API should look something like
```
def append(self, df: pa.Table, branch: str = MAIN_BRANCH)
...
def overwrite(self, df: pa.Table, overwrite_filter: BooleanExpression = ALWAYS_TRUE, branch: str = MAIN_BRANCH)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "vinjai (via GitHub)" <gi...@apache.org>.
vinjai commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2211703159
@kevinjqliu @Fokko
I can take this forward if no one is actively working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "Gowthami03B (via GitHub)" <gi...@apache.org>.
Gowthami03B commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2113233736
Hello, can I hop back on this train if no one else is actively working on this (again building off of kevin's work)? @kevinjqliu @Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "sl2902 (via GitHub)" <gi...@apache.org>.
sl2902 commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2585419953
@vinjai
Thanks for working on this feature
Is there any update regarding this feature? I would like to have this feature to perform a Write Audit Publish pattern in my pipeline. If it's still unavailable, is there any alternative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "vinjai (via GitHub)" <gi...@apache.org>.
vinjai commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2420730890
PR is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "dbuades (via GitHub)" <gi...@apache.org>.
dbuades commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2828024301
@sungwy @Fokko @kevinjqliu Sorry for the ping, I know you are very busy, but I noticed that the PR addressing this issue by @vinjai has been ready for review for a little while now: https://github.com/apache/iceberg-python/pull/941
We are really eager to use this feature, since it would allow us to finally move away from `pyspark` and fully adopt `pyiceberg`. Do you happen to have an estimate on when it might be reviewed and merged?
Thank you so much for all your work on this project !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "Gowthami03B (via GitHub)" <gi...@apache.org>.
Gowthami03B commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1947458069
I had an offline chat with @kevinjqliu , I shall work on this to build off of the PR created by kevin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2224059233
@vinjai yes! please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-2371644536
Hi @vinjai thank you very much for working on this issue. I'm just working through the list of open items to check if they are still actively being worked on. Are you still interested in contributing this feature to PyIceberg? 🧊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913306420
also note `dev/provision.py` which is used for integration tests already have statements to create tags and branchs
https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/dev/provision.py#L146-L148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #306:
URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1913304752
First pass, just refactoring
#312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support writing to a branch [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #306: Support writing to a branch
URL: https://github.com/apache/iceberg-python/issues/306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org