You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "greenlaw (via GitHub)" <gi...@apache.org> on 2025/09/04 20:03:27 UTC
[I] Update add_files operation to support branches [iceberg-python]
greenlaw opened a new issue, #2428:
URL: https://github.com/apache/iceberg-python/issues/2428
### Feature Request / Improvement
Write-to-branch functionality was added in #941, but it appears that support for this functionality in `add_files` operations was notably missing. It would be great if this workflow could be supported in the same way as other write operations.
I also dropped a message about this in the [slack channel](https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1757012415903679) about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "vinjai (via GitHub)" <gi...@apache.org>.
vinjai commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3301047792
Hey @greenlaw,
Thanks a lot for pointing this out 🙏. I had initially skipped adding this feature since the function is similar with the Spark implementation of procedure: [add_files](https://iceberg.apache.org/docs/nightly/spark-procedures/#add_files)
That said, I now realize there is not a straightforward way to achieve this in PyIceberg the way it is possible in the Java implementation. Your point makes complete sense and it would be valuable to add this as a feature.
I will raise a PR shortly. It should be a quick fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #2428: Update add_files operation to support branches
URL: https://github.com/apache/iceberg-python/issues/2428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3256644711
Thanks for opening this issue. Makes sense to me to allow the add_files operation to support branches.
I think we currently have limited capability to merge a branch back to main, just so you are aware of that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "greenlaw (via GitHub)" <gi...@apache.org>.
greenlaw commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3301127336
@vinjai PR would be greatly appreciated!
It seemed pretty straightforward to wire it up with the snapshot update class, so I started on my own PR but ran into some issues running integration tests on my machine and haven't gotten back to it yet. Hopefully I can straighten out my dev workflow and contribute in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "ForeverAngry (via GitHub)" <gi...@apache.org>.
ForeverAngry commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3256027325
Thats a pretty good question! Do you know, does the JAVA api support this? Ill take a look as well, when i get a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "greenlaw (via GitHub)" <gi...@apache.org>.
greenlaw commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3325233242
Awesome, thanks so much @vinjai !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "greenlaw (via GitHub)" <gi...@apache.org>.
greenlaw commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3256631431
Yes, it does. With the Java API I'm able to use [AppendFiles](https://iceberg.apache.org/javadoc/latest/org/apache/iceberg/AppendFiles.html), which supports [toBranch()](https://iceberg.apache.org/javadoc/latest/org/apache/iceberg/SnapshotUpdate.html#toBranch(java.lang.String)) along with [DataFiles.builder](https://iceberg.apache.org/javadoc/latest/org/apache/iceberg/DataFiles.Builder.html)
The code looks something like:
```
AppendFiles append = table.newAppend().toBranch("my_branch");
InputFile inputFile = HadoopInputFile.fromLocation(uri, hadoopConf);
Metrics metrics = ParquetUtil.fileMetrics(inputFile, metricsConfig);
DataFile file = DataFiles.builder(table.spec())
.withPath(path)
.withFormat(FileFormat.PARQUET)
.withFileSizeInBytes(inputFile.getLength())
.withMetrics(metrics)
.build();
append.appendFile(file);
append.commit();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "vinjai (via GitHub)" <gi...@apache.org>.
vinjai commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3305500706
Hey @greenlaw
Have raised the PR. Hope this addresses your concerns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Update add_files operation to support branches [iceberg-python]
Posted by "greenlaw (via GitHub)" <gi...@apache.org>.
greenlaw commented on issue #2428:
URL: https://github.com/apache/iceberg-python/issues/2428#issuecomment-3258774375
@kevinjqliu Thanks for the heads up! For my use case, I don't actually care about merging branches - I just need the ability to create and evolve divergent branches on a single table from a common ancestor.
With add_files branching support, plus the snapshot expiration capabilities in the upcoming 0.10.0 release, I think that's all I would need to move away from the Java API, although I'm still trying to identify any other feature gaps, so I appreciate your comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org