You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "3pacccccc (via GitHub)" <gi...@apache.org> on 2025/12/09 13:59:32 UTC
[PR] [improve][broker] Use atomic counter for ongoing transaction count [pulsar]
3pacccccc opened a new pull request, #25053:
URL: https://github.com/apache/pulsar/pull/25053
### Motivation
Currently, `txnMetaMap.size()` is used to determine if a new transaction can be added and for metrics reporting. However, `txnMetaMap` is a `ConcurrentSkipListMap`, and calling `size()` is an O(n) operation, which can consume significant system resources, especially under high transaction loads.
This PR replaces the use of `txnMetaMap.size()` with a dedicated atomic counter (`onGoingTxnCount`) to track the number of ongoing transactions in constant time (O(1)). This improves performance and reduces overhead in transaction management.
### Modifications
1. Introduced a `LongAdder` field `onGoingTxnCount` in `MLTransactionMetadataStore` to maintain the ongoing transaction count.
2. Updated all relevant methods to increment/decrement the counter when transactions are added or removed.
3. Refactored test cases to use `getOnGoingTxnCount()` instead of accessing the map via reflection, improving test clarity and maintainability.
4. Ensured metrics and coordinator stats now use the atomic counter instead of `txnMetaMap.size()`.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/3pacccccc/pulsar/pull/34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker] Use atomic counter for ongoing transaction count [pulsar]
Posted by "3pacccccc (via GitHub)" <gi...@apache.org>.
3pacccccc commented on code in PR #25053:
URL: https://github.com/apache/pulsar/pull/25053#discussion_r2606338122
##########
pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionMetadataStore.java:
##########
@@ -423,6 +434,7 @@ public CompletableFuture<Void> updateTxnStatus(TxnID txnID, TxnStatus newStatus,
abortedTransactionCount.increment();
}
txnMetaMap.remove(txnID.getLeastSigBits());
+ onGoingTxnCount.decrement();
Review Comment:
Thanks for catching this! It's indeed a valid race condition concern. I've updated the code accordingly. PTAL when you have a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker] Use atomic counter for ongoing transaction count [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #25053:
URL: https://github.com/apache/pulsar/pull/25053#discussion_r2605598835
##########
pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionMetadataStore.java:
##########
@@ -423,6 +434,7 @@ public CompletableFuture<Void> updateTxnStatus(TxnID txnID, TxnStatus newStatus,
abortedTransactionCount.increment();
}
txnMetaMap.remove(txnID.getLeastSigBits());
+ onGoingTxnCount.decrement();
Review Comment:
Are race conditions possible? just wondering if there should be a check that the transaction really was removed once?
Something like this:
```java
if (txnMetaMap.remove(txnID.getLeastSigBits()) != null) {
onGoingTxnCount.decrement();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker] Use atomic counter for ongoing transaction count [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25053:
URL: https://github.com/apache/pulsar/pull/25053#issuecomment-3635450837
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/25053?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 74.33%. Comparing base ([`270120c`](https://app.codecov.io/gh/apache/pulsar/commit/270120ce6e33e5a084397ca31186f1bb87835e48?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dd2ed37`](https://app.codecov.io/gh/apache/pulsar/commit/dd2ed37da18220382d1a6fd91f0a4ba239384d32?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/25053?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #25053 +/- ##
============================================
- Coverage 74.48% 74.33% -0.15%
+ Complexity 34186 34110 -76
============================================
Files 1921 1921
Lines 150376 150383 +7
Branches 17469 17469
============================================
- Hits 112005 111792 -213
- Misses 29499 29682 +183
- Partials 8872 8909 +37
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/25053/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/25053/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.68% <25.00%> (+0.33%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/25053/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.87% <0.00%> (-0.05%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/25053/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.86% <100.00%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/25053?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...n/coordinator/impl/MLTransactionMetadataStore.java](https://app.codecov.io/gh/apache/pulsar/pull/25053?src=pr&el=tree&filepath=pulsar-transaction%2Fcoordinator%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Ftransaction%2Fcoordinator%2Fimpl%2FMLTransactionMetadataStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLXRyYW5zYWN0aW9uL2Nvb3JkaW5hdG9yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvdHJhbnNhY3Rpb24vY29vcmRpbmF0b3IvaW1wbC9NTFRyYW5zYWN0aW9uTWV0YWRhdGFTdG9yZS5qYXZh) | `83.33% <100.00%> (+1.59%)` | :arrow_up: |
... and [106 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/25053/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker] Use atomic counter for ongoing transaction count [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- merged PR #25053:
URL: https://github.com/apache/pulsar/pull/25053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org