You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "damumu0625 (via GitHub)" <gi...@apache.org> on 2023/10/30 08:58:51 UTC
[PR] [flink] Support setting upper limit for inferred of source parallelism [incubator-paimon]
damumu0625 opened a new pull request, #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #2205
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support setting upper limit for inferring scan parallelism for source [incubator-paimon]
Posted by "damumu0625 (via GitHub)" <gi...@apache.org>.
damumu0625 commented on code in PR #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206#discussion_r1376152080
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -120,6 +120,13 @@ public class FlinkConnectorOptions {
"If it is false, parallelism of source are set by global parallelism."
+ " Otherwise, source parallelism is inferred from splits number (batch mode) or bucket number(streaming mode).");
+ public static final ConfigOption<Integer> INFER_SCAN_MAX_PARALLELISM =
+ ConfigOptions.key("scan.infer-parallelism.max")
+ .intType()
+ .defaultValue(Integer.MAX_VALUE)
Review Comment:
@JingsongLi I add this option to doc and the default value is set to 1024,please review again,thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support setting upper limit for inferring scan parallelism for source [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206#issuecomment-1786437548
@damumu0625 Please modify `FlinkConnectorOptions` default value too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support setting upper limit for inferring scan parallelism for source [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206#discussion_r1375964577
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -120,6 +120,13 @@ public class FlinkConnectorOptions {
"If it is false, parallelism of source are set by global parallelism."
+ " Otherwise, source parallelism is inferred from splits number (batch mode) or bucket number(streaming mode).");
+ public static final ConfigOption<Integer> INFER_SCAN_MAX_PARALLELISM =
+ ConfigOptions.key("scan.infer-parallelism.max")
+ .intType()
+ .defaultValue(Integer.MAX_VALUE)
Review Comment:
default 1000?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support setting upper limit for inferring scan parallelism for source [incubator-paimon]
Posted by "damumu0625 (via GitHub)" <gi...@apache.org>.
damumu0625 commented on code in PR #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206#discussion_r1376152080
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -120,6 +120,13 @@ public class FlinkConnectorOptions {
"If it is false, parallelism of source are set by global parallelism."
+ " Otherwise, source parallelism is inferred from splits number (batch mode) or bucket number(streaming mode).");
+ public static final ConfigOption<Integer> INFER_SCAN_MAX_PARALLELISM =
+ ConfigOptions.key("scan.infer-parallelism.max")
+ .intType()
+ .defaultValue(Integer.MAX_VALUE)
Review Comment:
The default value is set to 1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support setting upper limit for inferring scan parallelism for source [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2206:
URL: https://github.com/apache/incubator-paimon/pull/2206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org