You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Technoboy- (via GitHub)" <gi...@apache.org> on 2024/10/28 13:39:40 UTC
[PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Technoboy- opened a new pull request, #23524:
URL: https://github.com/apache/pulsar/pull/23524
### Motivation
If the `estimatedReadSize` larger than the `managedLedgerMaxReadsInFlightSizeInMB`, reading entries will fail.
```
2024-10-10T14:31:46,939+0000 [BookKeeperClientWorker-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheImpl - Time-out elapsed while acquiring enough permits on the memory limiter to read from ledger 114226, public/default/persistent/test-partition-0, estimated read size 120875300 bytes for 100 entries (check managedLedgerMaxReadsInFlightSizeInMB)
```
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#issuecomment-2461686043
@Technoboy- I'm working on a broader refactoring to address the issue. There are multiple challenges. I have reported multiple issues, #23482, #23504, #23505 and #23506. I'm working to resolve them soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#issuecomment-2461806234
One reason why the read size could exceed managedLedgerMaxReadsInFlightSizeInMB is #23482, that's what I'm also addressing in my changes that are WIP ([example commit](https://github.com/lhotari/pulsar/commit/769911dcf6f4f82fd8e9a96208b3ce1f16fe92d6), part of [the WIP changes](https://github.com/apache/pulsar/compare/master...lhotari:pulsar:lh-fix-individual-read-deduplication-and-limiting)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#discussion_r1829076284
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/RangeEntryCacheImpl.java:
##########
@@ -362,6 +362,9 @@ private AsyncCallbacks.ReadEntriesCallback handlePendingReadsLimits(ReadHandle l
}
long estimatedReadSize = (1 + lastEntry - firstEntry)
* (estimatedEntrySize + BOOKKEEPER_READ_OVERHEAD_PER_ENTRY);
+ if (estimatedReadSize > pendingReadsLimiter.getMaxReadsInFlightSize()) {
+ pendingReadsLimiter.setMaxReadsInFlightSize(estimatedReadSize);
Review Comment:
Here we are mutating the configuration of a shared object.
We could log an error but in the end the administrator has to change the value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#discussion_r1832172545
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/InflightReadsLimiter.java:
##########
@@ -56,7 +58,9 @@ public class InflightReadsLimiter implements AutoCloseable {
.help("Available space for inflight data read from storage or cache")
.register();
- private final long maxReadsInFlightSize;
+ @Setter
+ @Getter
+ private long maxReadsInFlightSize;
Review Comment:
This variable should stay final
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/RangeEntryCacheImpl.java:
##########
@@ -362,6 +362,9 @@ private AsyncCallbacks.ReadEntriesCallback handlePendingReadsLimits(ReadHandle l
}
long estimatedReadSize = (1 + lastEntry - firstEntry)
* (estimatedEntrySize + BOOKKEEPER_READ_OVERHEAD_PER_ENTRY);
+ if (estimatedReadSize > pendingReadsLimiter.getMaxReadsInFlightSize()) {
+ pendingReadsLimiter.setMaxReadsInFlightSize(estimatedReadSize);
Review Comment:
I am sorry, I was not clear.
The point is not about adding only the logging.
But we cannot update the configuration dynamically here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed pull request #23524: [fix][broker] Fix reading entries failed due to max in-flight reading
URL: https://github.com/apache/pulsar/pull/23524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#issuecomment-2463573106
> @Technoboy- I'm working on a broader refactoring to address the issue. There are multiple challenges. I have reported multiple issues, #23482, #23504, #23505 and #23506. I'm working to resolve them soon. While fixing the issues, I have discovered more. For example, when reads get de-duplicated by PendingReadsManager and there are partial matches, the permits will get acquired also for the partial reads. I have a large change set in progress, which I will split into smaller pull requests once I have addressed the issues with the refactoring and the improvements and I have tests passing. Since my main goal is to improve caching for Key_Shared subscriptions, this has revealed more gaps in addressing that. In the current solution replay reads aren't cached at all. I noticed that @eolivelli has reported a related issue #16421 about that for Shared subscriptions. The comment [#16421 (comment)](https://github.com/apache/pulsar/issues/16421#issuecomment-1177516197) is relevant. Messages
in the replay queues shouldn't be discarded from the cache. I'm also trying to address that in my experiments. That's why the changes have expanded to also address broker cache short comings.
ok, i will close this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Fix reading entries failed due to max in-flight reading [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #23524:
URL: https://github.com/apache/pulsar/pull/23524#discussion_r1830281373
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/RangeEntryCacheImpl.java:
##########
@@ -362,6 +362,9 @@ private AsyncCallbacks.ReadEntriesCallback handlePendingReadsLimits(ReadHandle l
}
long estimatedReadSize = (1 + lastEntry - firstEntry)
* (estimatedEntrySize + BOOKKEEPER_READ_OVERHEAD_PER_ENTRY);
+ if (estimatedReadSize > pendingReadsLimiter.getMaxReadsInFlightSize()) {
+ pendingReadsLimiter.setMaxReadsInFlightSize(estimatedReadSize);
Review Comment:
address the comment.
Add log here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org