You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/12/20 03:36:20 UTC
[GitHub] [kafka] ijuma opened a new pull request, #13024: MINOR: Avoid unnecessary allocations in index binary search
ijuma opened a new pull request, #13024:
URL: https://github.com/apache/kafka/pull/13024
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1359582927
@chia7712 Fixed the bug and also tweaked the name of the enum and variable to avoid confusion with `SearchEntityType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
chia7712 commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1359594391
> Fixed the bug and also tweaked the name of the enum and variable to avoid confusion with SearchEntityType.
Are there failed tests related to that bug (inverse usage of SearchType) ? Not sure why I don't see the QA triggered by this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13024:
URL: https://github.com/apache/kafka/pull/13024#discussion_r1053450391
##########
storage/src/main/java/org/apache/kafka/server/log/internals/AbstractIndex.java:
##########
@@ -484,27 +478,35 @@ private static MappedByteBuffer createMappedBuffer(RandomAccessFile raf, boolean
}
/**
- * Lookup lower and upper bounds for the given target.
+ * Lookup lower or upper bounds for the given target.
*/
- private BinarySearchResult indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity) {
+ private int indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity,
+ SearchType searchType) {
// check if the index is empty
if (entries == 0)
- return new BinarySearchResult(-1, -1);
+ return -1;
int firstHotEntry = Math.max(0, entries - 1 - warmEntries());
// check if the target offset is in the warm section of the index
if (compareIndexEntry(parseEntry(idx, firstHotEntry), target, searchEntity) < 0) {
- return binarySearch(idx, target, searchEntity, firstHotEntry, entries - 1);
+ return binarySearch(idx, target, searchEntity, searchType, firstHotEntry, entries - 1);
}
// check if the target offset is smaller than the least offset
- if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0)
- return new BinarySearchResult(-1, 0);
+ if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0) {
+ switch (searchType) {
+ case SMALLEST_UPPER_BOUND:
+ return -1;
+ case LARGEST_LOWER_BOUND:
+ return 0;
+ }
Review Comment:
It's better to have exhaustive matches than to add a `default`. Unfortunately, the Java compiler doesn't always infer that the method always returns due to an exhaustive match (at least not with old school switches, the new switch probably handles it better) and hence we have to add the explicit `default`. So, I'd prefer to leave as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1360792320
Test failures are unrelated:
> Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsWithCustomForwardingAdminIntegrationTest.testReplicationIsCreatingTopicsUsingProvidedForwardingAdmin()
> Build / JDK 11 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testDelayedConfigurationOperations()
> Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testReplication()
> Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsWithCustomForwardingAdminIntegrationTest.testCreatePartitionsUseProvidedForwardingAdmin()
> Build / JDK 8 and Scala 2.12 / kafka.api.SslEndToEndAuthorizationTest.[2] quorum=kraft, isIdempotenceEnabled=false
> Build / JDK 8 and Scala 2.12 / kafka.api.TransactionsExpirationTest.testTransactionAfterProducerIdExpires(String).quorum=kraft
> Build / JDK 8 and Scala 2.12 / kafka.server.FetchRequestBetweenDifferentIbpTest.testControllerOldIBP()
> Build / JDK 8 and Scala 2.12 / org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.[1] true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1361298288
Annoyingly, the commit was somehow pushed prematurely while I was still editing the commit message - not quite sure how that happened. Apologies for the bad commit message and the missing reviewers line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1359643024
> Are there failed tests related to that bug (inverse usage of SearchType) ? Not sure why I don't see the QA triggered by this PR
I was wondering the same thing for both and am investigating. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1359675311
@chia7712 `OffsetIndexTest` does fail without the fix and pass with the fix. And the CI build is queued now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma merged pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma merged PR #13024:
URL: https://github.com/apache/kafka/pull/13024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a diff in pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
chia7712 commented on code in PR #13024:
URL: https://github.com/apache/kafka/pull/13024#discussion_r1053182775
##########
storage/src/main/java/org/apache/kafka/server/log/internals/AbstractIndex.java:
##########
@@ -484,27 +478,35 @@ private static MappedByteBuffer createMappedBuffer(RandomAccessFile raf, boolean
}
/**
- * Lookup lower and upper bounds for the given target.
+ * Lookup lower or upper bounds for the given target.
*/
- private BinarySearchResult indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity) {
+ private int indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity,
+ SearchType searchType) {
// check if the index is empty
if (entries == 0)
- return new BinarySearchResult(-1, -1);
+ return -1;
int firstHotEntry = Math.max(0, entries - 1 - warmEntries());
// check if the target offset is in the warm section of the index
if (compareIndexEntry(parseEntry(idx, firstHotEntry), target, searchEntity) < 0) {
- return binarySearch(idx, target, searchEntity, firstHotEntry, entries - 1);
+ return binarySearch(idx, target, searchEntity, searchType, firstHotEntry, entries - 1);
}
// check if the target offset is smaller than the least offset
- if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0)
- return new BinarySearchResult(-1, 0);
+ if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0) {
+ switch (searchType) {
+ case SMALLEST_UPPER_BOUND:
+ return -1;
+ case LARGEST_LOWER_BOUND:
+ return 0;
+ }
Review Comment:
How about throwing `IllegalStateException` from the default label for consistency?
##########
storage/src/main/java/org/apache/kafka/server/log/internals/AbstractIndex.java:
##########
@@ -484,27 +478,35 @@ private static MappedByteBuffer createMappedBuffer(RandomAccessFile raf, boolean
}
/**
- * Lookup lower and upper bounds for the given target.
+ * Lookup lower or upper bounds for the given target.
*/
- private BinarySearchResult indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity) {
+ private int indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity,
+ SearchType searchType) {
// check if the index is empty
if (entries == 0)
- return new BinarySearchResult(-1, -1);
+ return -1;
int firstHotEntry = Math.max(0, entries - 1 - warmEntries());
// check if the target offset is in the warm section of the index
if (compareIndexEntry(parseEntry(idx, firstHotEntry), target, searchEntity) < 0) {
- return binarySearch(idx, target, searchEntity, firstHotEntry, entries - 1);
+ return binarySearch(idx, target, searchEntity, searchType, firstHotEntry, entries - 1);
}
// check if the target offset is smaller than the least offset
- if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0)
- return new BinarySearchResult(-1, 0);
+ if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0) {
+ switch (searchType) {
+ case SMALLEST_UPPER_BOUND:
+ return -1;
Review Comment:
It seems to me `SMALLEST_UPPER_BOUND` should return 0 and `LARGEST_LOWER_BOUND` should return -1
##########
storage/src/main/java/org/apache/kafka/server/log/internals/AbstractIndex.java:
##########
@@ -517,13 +519,19 @@ private BinarySearchResult binarySearch(ByteBuffer idx, long target, IndexSearch
else if (compareResult < 0)
lo = mid;
else
- return new BinarySearchResult(mid, mid);
+ return mid;
+ }
+ switch (searchType) {
+ case SMALLEST_UPPER_BOUND:
Review Comment:
ditto. maybe we get the incorrect returned value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
divijvaidya commented on PR #13024:
URL: https://github.com/apache/kafka/pull/13024#issuecomment-1359348307
^ please ignore my approval. @chia7712's comments need to be fixed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13024: MINOR: Avoid unnecessary allocations in index binary search
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13024:
URL: https://github.com/apache/kafka/pull/13024#discussion_r1053451180
##########
storage/src/main/java/org/apache/kafka/server/log/internals/AbstractIndex.java:
##########
@@ -484,27 +478,35 @@ private static MappedByteBuffer createMappedBuffer(RandomAccessFile raf, boolean
}
/**
- * Lookup lower and upper bounds for the given target.
+ * Lookup lower or upper bounds for the given target.
*/
- private BinarySearchResult indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity) {
+ private int indexSlotRangeFor(ByteBuffer idx, long target, IndexSearchType searchEntity,
+ SearchType searchType) {
// check if the index is empty
if (entries == 0)
- return new BinarySearchResult(-1, -1);
+ return -1;
int firstHotEntry = Math.max(0, entries - 1 - warmEntries());
// check if the target offset is in the warm section of the index
if (compareIndexEntry(parseEntry(idx, firstHotEntry), target, searchEntity) < 0) {
- return binarySearch(idx, target, searchEntity, firstHotEntry, entries - 1);
+ return binarySearch(idx, target, searchEntity, searchType, firstHotEntry, entries - 1);
}
// check if the target offset is smaller than the least offset
- if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0)
- return new BinarySearchResult(-1, 0);
+ if (compareIndexEntry(parseEntry(idx, 0), target, searchEntity) > 0) {
+ switch (searchType) {
+ case SMALLEST_UPPER_BOUND:
+ return -1;
Review Comment:
Good catch, I reversed it here and in the other place. Will fix both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org