You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2023/11/23 03:24:04 UTC
[PR] [flink] Limit max split while continuously scanning to avoid JobManager OOM [incubator-paimon]
leaves12138 opened a new pull request, #2373:
URL: https://github.com/apache/incubator-paimon/pull/2373
### Purpose
close: https://github.com/apache/incubator-paimon/issues/2370
### Tests
Added test.
### API and Format
No other influence to user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Limit max split while continuously scanning to avoid JobManager OOM [incubator-paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2373:
URL: https://github.com/apache/incubator-paimon/pull/2373#discussion_r1402975791
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -219,6 +219,12 @@ public class CoreOptions implements Serializable {
.defaultValue(Duration.ofSeconds(10))
.withDescription("The discovery interval of continuous reading.");
+ public static final ConfigOption<Integer> SCAN_SPLIT_MAX_PER_TASK =
+ key("scan.split-max-per-task")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Limit max split while continuously scanning to avoid JobManager OOM [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2373:
URL: https://github.com/apache/incubator-paimon/pull/2373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Limit max split while continuously scanning to avoid JobManager OOM [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2373:
URL: https://github.com/apache/incubator-paimon/pull/2373#discussion_r1402942327
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -219,6 +219,12 @@ public class CoreOptions implements Serializable {
.defaultValue(Duration.ofSeconds(10))
.withDescription("The discovery interval of continuous reading.");
+ public static final ConfigOption<Integer> SCAN_SPLIT_MAX_PER_TASK =
+ key("scan.split-max-per-task")
Review Comment:
`scan.max-splits-per-task`?
And the description should be more...
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/ContinuousFileSplitEnumerator.java:
##########
@@ -85,7 +87,8 @@ public ContinuousFileSplitEnumerator(
@Nullable Long nextSnapshotId,
long discoveryInterval,
StreamTableScan scan,
- BucketMode bucketMode) {
+ BucketMode bucketMode,
+ Integer splitMaxPerTask) {
Review Comment:
why this is Integer instead of int?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Limit max split while continuously scanning to avoid JobManager OOM [incubator-paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2373:
URL: https://github.com/apache/incubator-paimon/pull/2373#discussion_r1402976054
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/ContinuousFileSplitEnumerator.java:
##########
@@ -85,7 +87,8 @@ public ContinuousFileSplitEnumerator(
@Nullable Long nextSnapshotId,
long discoveryInterval,
StreamTableScan scan,
- BucketMode bucketMode) {
+ BucketMode bucketMode,
+ Integer splitMaxPerTask) {
Review Comment:
used int instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org