You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lipenglin (via GitHub)" <gi...@apache.org> on 2023/09/16 09:51:11 UTC
[GitHub] [incubator-paimon] lipenglin opened a new pull request, #2033: [core] Fix outputlevel is still 0 in extreme scenarios
lipenglin opened a new pull request, #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Fix outputlevel is still 0 in extreme scenarios.
This may not be a common case, but it is indeed possible.
For a newly created paimon pk table, with default parameters for num-levels and
others, without adjusting the max-size-amp, size-ratio, and other
configurations, assuming all configurations are using default values.
For a specific bucket:
file1 (1kb) is written,
file2 (1kb) is written,
file3 (1kb) is written,
file4 (1mb) is written,
file5 (10mb) is written
The srS of level0 will be sorted according to
https://github.com/apache/incubator-paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/mergetree/Levels.java#L58
In this situation, these five level0 files may not go through the pickForSizeAmp
function, outputlevel is 0. This may cause a larger file to appear in level0. I
think setting the outputlevel to 5 would be more ideal.
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
Add UT org.apache.paimon.mergetree.compact.UniversalCompactionTest#testExtremeCaseNoOutputLevel0
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1723305958
@lipenglin You are right, see above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi merged pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1724737316
Please fix tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] lipenglin closed pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "lipenglin (via GitHub)" <gi...@apache.org>.
lipenglin closed pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
URL: https://github.com/apache/incubator-paimon/pull/2033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] lipenglin commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "lipenglin (via GitHub)" <gi...@apache.org>.
lipenglin commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1723318880
> @lipenglin You are right, see above comment.
done.
PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] lipenglin commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "lipenglin (via GitHub)" <gi...@apache.org>.
lipenglin commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1723144996
> See `UniversalCompaction.createUnit`, we force picking to output level 1...
But actually, this result is 0.
` UniversalCompaction compaction = new UniversalCompaction(200, 1, 5);
Optional<CompactUnit> pick =
compaction.pick(
6,
Arrays.asList(
level(0, 1),
level(0, 1),
level(0, 1),
level(0, 1024),
level(0, 1024 * 1024)));
assertThat(pick.isPresent()).isTrue();
long[] results = pick.get().files().stream().mapToLong(DataFileMeta::fileSize).toArray();
assertThat(results).isEqualTo(new long[] {1, 1, 1, 1024, 1024 * 1024});
// pick.get().outputLevel() -> the result is 0
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#discussion_r1328646525
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/UniversalCompaction.java:
##########
@@ -171,6 +171,10 @@ static CompactUnit createUnit(List<LevelSortedRun> runs, int maxLevel, int runCo
}
}
+ if (outputLevel == 0) {
Review Comment:
```
if (runCount == runs.size()) {
outputLevel = maxLevel;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] lipenglin commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "lipenglin (via GitHub)" <gi...@apache.org>.
lipenglin commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1724760916
> Please fix tests.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on pull request #2033: [core] Fix outputlevel is still 0 in extreme scenarios
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2033:
URL: https://github.com/apache/incubator-paimon/pull/2033#issuecomment-1723120240
See `UniversalCompaction.createUnit`, we force picking to output level 1...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org