You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "czy006 (via GitHub)" <gi...@apache.org> on 2026/08/17 09:06:48 UTC
[PR] [hotfix][Optimizer] Guard quota calculations against zero divisors [amoro]
czy006 opened a new pull request, #4324:
URL: https://github.com/apache/amoro/pull/4324
## Brief change log
Clamp per-process quota and target quota values to at least one, and handle non-positive fragment ratios, preventing zero-division failures in optimizer quota calculations.
## How was this patch tested?
- [x] Add configuration test coverage for zero and non-positive guard cases.
- [x] Add screenshots for manual tests if appropriate (not applicable: backend-only change).
- [x] Run `TestOptimizingConfig` locally with JDK 11 before creating this pull request.
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3802917866
##########
amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java:
##########
@@ -104,7 +105,8 @@ private TableProperties() {}
public static final long SELF_OPTIMIZING_MAX_TASK_SIZE_DEFAULT = 134217728; // 128 MB
public static final String SELF_OPTIMIZING_FRAGMENT_RATIO = "self-optimizing.fragment-ratio";
- public static final int SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT = 8;
+ public static final int SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT =
Review Comment:
```suggestion
public static final int SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT = 8;
```
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -26,6 +26,8 @@
@JsonIgnoreProperties(ignoreUnknown = true)
public class OptimizingConfig {
+ public static final int DEFAULT_FRAGMENT_RATIO = 8;
Review Comment:
Why define the defaut value there
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
```suggestion
return targetSize / Math.max(1, fragmentRatio);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3802571088
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +226,9 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ // A non-positive fragment ratio (misconfigured property or legacy persisted value) means
+ // "no fragmenting": fall back to the target size instead of dividing by zero in planners.
+ return fragmentRatio <= 0 ? targetSize : targetSize / fragmentRatio;
Review Comment:
Good catch. I updated the fallback to use the documented default fragment ratio of 8, so the default 128 MiB target size yields a 16 MiB fragment threshold. CommonPartitionEvaluator now calls OptimizingConfig.maxFragmentSize() instead of performing the raw division, and a planner-level regression test covers both zero and negative ratios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3804181871
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -26,6 +26,8 @@
@JsonIgnoreProperties(ignoreUnknown = true)
public class OptimizingConfig {
+ public static final int DEFAULT_FRAGMENT_RATIO = 8;
Review Comment:
Agreed. I removed the default constant from `OptimizingConfig`; the configured default remains owned by `TableProperties`.
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
Thanks. Following the later setter-validation suggestion, the lower bound is now applied once in `setFragmentRatio()` instead of inside `maxFragmentSize()`.
##########
amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java:
##########
@@ -104,7 +105,8 @@ private TableProperties() {}
public static final long SELF_OPTIMIZING_MAX_TASK_SIZE_DEFAULT = 134217728; // 128 MB
public static final String SELF_OPTIMIZING_FRAGMENT_RATIO = "self-optimizing.fragment-ratio";
- public static final int SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT = 8;
+ public static final int SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT =
Review Comment:
Done. `SELF_OPTIMIZING_FRAGMENT_RATIO_DEFAULT` is restored to `8` in `TableProperties`.
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
Agreed and updated. `setFragmentRatio()` now normalizes zero and negative values to `1`, and `maxFragmentSize()` remains a simple derived calculation. The evaluator also reuses this method.
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
I have not added the upper bound in this hotfix. `fragmentRatio` is dimensionless while `targetSize` is a byte size, so this would introduce a new cross-field invariant. Also, `TableConfigurations` currently calls `setFragmentRatio()` before `setTargetSize()`, so the setter cannot validate against the final target size reliably. A very large positive ratio produces a zero fragment threshold but does not cause the division failure addressed here. I suggest handling an upper-bound policy separately, with defined semantics and validation after both values are available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard quota calculations against zero divisors [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3800693563
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +226,9 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ // A non-positive fragment ratio (misconfigured property or legacy persisted value) means
+ // "no fragmenting": fall back to the target size instead of dividing by zero in planners.
+ return fragmentRatio <= 0 ? targetSize : targetSize / fragmentRatio;
Review Comment:
the default `target-size` is 134217728 which is too large
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3802929049
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +226,9 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ // A non-positive fragment ratio (misconfigured property or legacy persisted value) means
+ // "no fragmenting": fall back to the target size instead of dividing by zero in planners.
+ return fragmentRatio <= 0 ? targetSize : targetSize / fragmentRatio;
Review Comment:
ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3802956342
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
I believe the param would be validated in `setFragmentRatio` instead of getter method. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4324:
URL: https://github.com/apache/amoro/pull/4324#discussion_r3802947932
##########
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java:
##########
@@ -226,7 +228,8 @@ public OptimizingConfig setMinTargetSizeRatio(double minTargetSizeRatio) {
}
public long maxFragmentSize() {
- return targetSize / fragmentRatio;
+ int effectiveFragmentRatio = fragmentRatio > 0 ? fragmentRatio : DEFAULT_FRAGMENT_RATIO;
+ return targetSize / effectiveFragmentRatio;
Review Comment:
I think we can also add a upper limitation for fragmentRatio. it should be less or equals than `targetSize`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix][Optimizer] Guard invalid fragment ratio and target quota [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #4324:
URL: https://github.com/apache/amoro/pull/4324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org