You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "liangyepianzhou (via GitHub)" <gi...@apache.org> on 2023/02/24 08:47:30 UTC
[GitHub] [pulsar] liangyepianzhou opened a new pull request, #19631: [improve][txn] PIP-250: Add snapshot stats in TransactionBufferStats
liangyepianzhou opened a new pull request, #19631:
URL: https://github.com/apache/pulsar/pull/19631
Master https://github.com/apache/pulsar/issues/19628
### Motivation
Modify the admin tool to get the stats of the transaction buffer snapshot segment when the transaction buffer snapshot segment is enabled.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [x] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/liangyepianzhou/pulsar/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1227929917
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
I believe the most crucial consideration is whether we need to provide full concurrency safety guarantees for this admin API. If it is necessary to ensure that the data obtained by the admin is not affected by concurrent operations, then we must introduce locks for the `generateSnapshotStats` and `putAbortedTxnAndPosition` methods. We also need to consider whether we should introduce locks for the hot path of message sending in this admin API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#issuecomment-1491180167
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1227966204
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
I can replace ` snapshotStats.segmentsSize = this.indexes.size();` with `snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();` which update before persistenting segment, but the concurrent problem still exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1119561849
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/AbortedTxnProcessor.java:
##########
@@ -65,10 +66,9 @@ public interface AbortedTxnProcessor {
CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition);
/**
- * Get the lastSnapshotTimestamps.
- * @return the lastSnapshotTimestamps.
+ * Get the snapshot stats form the processor.
*/
- long getLastSnapshotTimestamps();
+ void getSnapshotStats(TransactionBufferStats stats);
Review Comment:
The implementation of this method is to set the property of stat. Why does it named `getxxx`?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
If there has an in flight task which is typed `write segment`, there is an issue:
- the variable `unsealedAbortTxnIDs` will be zero
- the variable `segmentsSize` is one less than the correct value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1226000303
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
In my view, this particular call is only retrieving data at the lastSnapshotTimestamps timestamp. If the user has an expected result, they can make multiple admin calls for that purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1227858172
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
Maybe they can make some judgments by the `lastSnapshotTimestamps` timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou closed pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou closed pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
URL: https://github.com/apache/pulsar/pull/19631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19631: [improve][txn] PIP-251: Add snapshot stats in TransactionBufferStats
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19631:
URL: https://github.com/apache/pulsar/pull/19631#discussion_r1226210982
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -384,8 +385,12 @@ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
}
@Override
- public long getLastSnapshotTimestamps() {
- return this.lastSnapshotTimestamps;
+ public void getSnapshotStats(TransactionBufferStats stats) {
+ stats.lastSnapshotTimestamps = this.lastSnapshotTimestamps;
+ TransactionBufferStats.SnapshotStats snapshotStats = new TransactionBufferStats.SnapshotStats();
+ snapshotStats.segmentsSize = this.indexes.size();
+ snapshotStats.unsealedAbortTxnIDs = this.unsealedTxnIds.size();
+ stats.snapshotStats = snapshotStats;
Review Comment:
> In my view, this particular call is only retrieving data at the lastSnapshotTimestamps timestamp. If the user has an expected result, they can make multiple admin calls for that purpose.
If users knew the correct value, they do not need this API anymore, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org