You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "clairemcginty (via GitHub)" <gi...@apache.org> on 2026/07/21 15:07:39 UTC
[PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
clairemcginty opened a new pull request, #39408:
URL: https://github.com/apache/beam/pull/39408
https://github.com/apache/beam/pull/38269 added SortOrder support to Iceberg writes via Managed IO, but it's not accessible through IcebergIO directly. This PR adds support for setting SortOrder+PartitionSpec through the IcebergIO.writeRows builder.
It does make serialization of the transform more complex which might be antithetical to the goal of keeping new transforms as portable as possible -- cc @ahmedabu98, let me know if not exposing these params via `IcebergIO` was by design and if I should just switch to ManagedIO if I need to specify.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on PR #39408:
URL: https://github.com/apache/beam/pull/39408#issuecomment-5095342301
> Sorry was OOO for some time and just seeing this now. I think this PR is a good addition, and doesn’t harm the portable case at all. Just had one nit regarding naming but it LGTM! Feel free to convert it so we can run some tests against it
>
> Also I’m curious to know if the ManagedIO route is causing any friction for your use case?
sounds good! Just renamed the method and undrafted the PR.
No, so far so good! The iceberg write API in Scio goes through ManagedIO and works well. I've just been parity testing a lot of iceberg IO features on both managedIO + icebergIO and caught this particular discrepancy!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #39408:
URL: https://github.com/apache/beam/pull/39408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #39408:
URL: https://github.com/apache/beam/pull/39408#issuecomment-5095322602
> [!CAUTION]
> The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
Posted by "clairemcginty (via GitHub)" <gi...@apache.org>.
clairemcginty commented on code in PR #39408:
URL: https://github.com/apache/beam/pull/39408#discussion_r3659939598
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java:
##########
@@ -483,6 +491,26 @@ public WriteRows withWriteProperties(Map<String, String> writeProperties) {
return toBuilder().setWriteProperties(writeProperties).build();
}
+ /**
+ * Defines the desired Partition Spec to be applied when the Iceberg table must be dynamically
+ * created, e.g. `bucket(id_field, 32)` or `day(timestamp_field)`
+ *
+ * <p>See: https://iceberg.apache.org/spec/#partitioning
+ */
+ public WriteRows withPartitionSpec(List<String> partitionFields) {
Review Comment:
sounds good! updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39408:
URL: https://github.com/apache/beam/pull/39408#discussion_r3659599835
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java:
##########
@@ -483,6 +491,26 @@ public WriteRows withWriteProperties(Map<String, String> writeProperties) {
return toBuilder().setWriteProperties(writeProperties).build();
}
+ /**
+ * Defines the desired Partition Spec to be applied when the Iceberg table must be dynamically
+ * created, e.g. `bucket(id_field, 32)` or `day(timestamp_field)`
+ *
+ * <p>See: https://iceberg.apache.org/spec/#partitioning
+ */
+ public WriteRows withPartitionSpec(List<String> partitionFields) {
Review Comment:
nit: maybe more accurate to name this `withPartitionFields()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org