You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2025/09/26 15:13:58 UTC
[PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
mxm opened a new pull request, #14191:
URL: https://github.com/apache/iceberg/pull/14191
This adds the option to automatically set the write parallelism to the max available write parallelism, i.e. the job parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Document writeParallelism and fail on invalid configuration [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #14191:
URL: https://github.com/apache/iceberg/pull/14191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#discussion_r2385215493
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java:
##########
@@ -39,6 +39,18 @@ public class DynamicRecord {
private boolean upsertMode;
@Nullable private Set<String> equalityFields;
+ /**
+ * Constructs a new DynamicRecord.
+ *
+ * @param tableIdentifier The target table identifier.
+ * @param branch The target table branch.
+ * @param schema The target table schema.
+ * @param rowData The data matching the provided schema.
+ * @param partitionSpec The target table {@link PartitionSpec}.
+ * @param distributionMode The {@link DistributionMode}.
+ * @param writeParallelism The number of parallel writers. If set to {@literal <= 0}, will
Review Comment:
What if we set it to `MAX_INT`. It is easier to calculate with `min` later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Document writeParallelism and fail on invalid configuration [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#issuecomment-3611073160
Merged to main.
Thanks @mxm for the PR, and @Guosmilesmile for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#discussion_r2384436588
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicSinkUtil.java:
##########
@@ -62,4 +62,14 @@ static int safeAbs(int input) {
return -input;
}
+
+ static int firstPositive(int first, int second) {
+ if (first > 0) {
+ return first;
+ }
Review Comment:
nit: new line.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicSinkUtil.java:
##########
@@ -62,4 +62,14 @@ static int safeAbs(int input) {
return -input;
}
+
+ static int firstPositive(int first, int second) {
+ if (first > 0) {
+ return first;
+ }
+ if (second > 0) {
+ return second;
+ }
+ throw new IllegalArgumentException("None of the supplied ints were positive!");
Review Comment:
Should we print out both values to facilitate troubleshooting?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicSinkUtil.java:
##########
@@ -62,4 +62,14 @@ static int safeAbs(int input) {
return -input;
}
+
+ static int firstPositive(int first, int second) {
+ if (first > 0) {
+ return first;
+ }
+ if (second > 0) {
+ return second;
+ }
Review Comment:
nit: new line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14191: Flink: Dynamic Sink: Use max write parallelism if none is supplied
URL: https://github.com/apache/iceberg/pull/14191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#issuecomment-3459152836
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#issuecomment-3494205462
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Use max write parallelism if none is supplied [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#discussion_r2513444892
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java:
##########
@@ -39,6 +39,18 @@ public class DynamicRecord {
private boolean upsertMode;
@Nullable private Set<String> equalityFields;
+ /**
+ * Constructs a new DynamicRecord.
+ *
+ * @param tableIdentifier The target table identifier.
+ * @param branch The target table branch.
+ * @param schema The target table schema.
+ * @param rowData The data matching the provided schema.
+ * @param partitionSpec The target table {@link PartitionSpec}.
+ * @param distributionMode The {@link DistributionMode}.
+ * @param writeParallelism The number of parallel writers. If set to {@literal <= 0}, will
Review Comment:
I'm ok with using Integer.MAX_VALUE, but I wouldn't do it simply because it is easier to calculate. Actually, using Integer.MAX_VALUE already works because we always cap the write parallelism at the job parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Dynamic Sink: Document writeParallelism and fail on invalid configuration [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #14191:
URL: https://github.com/apache/iceberg/pull/14191#issuecomment-3607254344
I've reworked the PR around the current behavior to cap the write parallelism at the max write parallelism. I've documented the behavior and added a check to fail on invalid write parallelism settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org