You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/20 07:15:43 UTC
[GitHub] [pulsar] liangyepianzhou opened a new pull request, #15682: [improve][txn] Add a admin tool to check message pending ack stats
liangyepianzhou opened a new pull request, #15682:
URL: https://github.com/apache/pulsar/pull/15682
### Motivation
When a message is acknowledged by a transaction, it is in pending ack stats until the end of the transaction.
But we have no way to judge whether the message is in the pendingAck state on the client side.
### Modification
Add a admin tool to check message pending ack stats.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `no-need-doc`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-added`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on code in PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#discussion_r905681190
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -342,4 +343,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
Review Comment:
```suggestion
@Path("/pendingAckStats/{tenant}/{namespace}/{topic}/{subName}")
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -342,4 +343,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
+ @ApiOperation(value = "Get position stats in pending ack.")
+ @ApiResponses(value = {@ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Tenant or cluster or namespace or topic "
+ + "or subscription name doesn't exist"),
+ @ApiResponse(code = 503, message = "This Broker is not configured "
+ + "with transactionCoordinatorEnabled=true."),
+ @ApiResponse(code = 307, message = "Topic is not owned by this broker!"),
+ @ApiResponse(code = 405, message = "Pending ack handle don't use managedLedger!"),
+ @ApiResponse(code = 400, message = "Topic is not a persistent topic!"),
+ @ApiResponse(code = 409, message = "Concurrent modification")})
+ public void getPositionStatsInPendingAck(@Suspended final AsyncResponse asyncResponse,
+ @QueryParam("authoritative")
+ @DefaultValue("false") boolean authoritative,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @PathParam("subName") String subName,
+ @QueryParam("ledgerId") Long ledgerId,
Review Comment:
Maybe we could add the params `ledgerId` and `entryId` in the path, it's more rest style, and we don't need to validate them.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/v3/AdminApiTransactionTest.java:
##########
@@ -613,6 +619,117 @@ public void testUpdateTransactionCoordinatorNumber() throws Exception {
}
}
+
+ @Test
+ public void testCheckPositionInPendingAckState() throws Exception {
+ String topic = "persistent://public/default/test";
+ String subName = "sub";
+ initTransaction(1);
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.SECONDS)
+ .build()
+ .get();
+
+ @Cleanup
+ Producer<byte[]> producer = pulsarClient.newProducer()
+ .sendTimeout(5, TimeUnit.SECONDS)
+ .enableBatching(false)
+ .topic(topic)
+ .create();
+ @Cleanup
+ Consumer<byte[]> consumer = pulsarClient.newConsumer()
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscribe();
+
+ producer.newMessage().send();
+
+ Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+ MessageIdImpl messageId = (MessageIdImpl) message.getMessageId();
+
+ PositionInPendingAckStats result = admin.transactions().checkPositionInPendingAckState(topic, subName,
+ messageId.getLedgerId(), messageId.getEntryId(), null);
+ assertEquals(result.state, PositionInPendingAckStats.State.PendingAckNotReady);
Review Comment:
```suggestion
assertEquals(result.state, PositionInPendingAckStats.State.PendingAckNotReady);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#issuecomment-1147338773
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#discussion_r887626615
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -339,4 +340,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
+ @ApiOperation(value = "Get position stats in pending ack.")
+ @ApiResponses(value = {@ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Tenant or cluster or namespace or topic "
+ + "or subscription name doesn't exist"),
+ @ApiResponse(code = 503, message = "This Broker is not configured "
+ + "with transactionCoordinatorEnabled=true."),
+ @ApiResponse(code = 307, message = "Topic is not owned by this broker!"),
+ @ApiResponse(code = 405, message = "Pending ack handle don't use managedLedger!"),
+ @ApiResponse(code = 400, message = "Topic is not a persistent topic!"),
+ @ApiResponse(code = 409, message = "Concurrent modification")})
+ public void getPositionStatsInPendingAck(@Suspended final AsyncResponse asyncResponse,
+ @QueryParam("authoritative")
+ @DefaultValue("false") boolean authoritative,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @PathParam("subName") String subName,
+ @QueryParam("position") String position) {
+ try {
+ checkArgument(position != null, "Message position should not be null.");
+ checkTransactionCoordinatorEnabled();
+ validateTopicName(tenant, namespace, encodedTopic);
+ PositionImpl positionImpl = PositionImpl.parsePositionFromString(position);
+ internalGetPositionStatsPendingAckStats(authoritative, subName, positionImpl)
+ .thenAccept(positionInPendingAckStats -> asyncResponse.resume(positionInPendingAckStats))
+ .exceptionally(ex -> {
+ log.warn("{} Failed to check position [{}] stats for topic [{}], subscription [{}]",
+ clientAppId(), position, topicName, subName, ex);
+ resumeAsyncResponseExceptionally(asyncResponse, ex);
+ return null;
+ });
+ } catch (Exception ex) {
+ log.warn("Failed to get position stats in pending ack");
+ resumeAsyncResponseExceptionally(asyncResponse, ex);
+ }
+
+
+
Review Comment:
delete
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -339,4 +340,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
+ @ApiOperation(value = "Get position stats in pending ack.")
+ @ApiResponses(value = {@ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Tenant or cluster or namespace or topic "
+ + "or subscription name doesn't exist"),
+ @ApiResponse(code = 503, message = "This Broker is not configured "
+ + "with transactionCoordinatorEnabled=true."),
+ @ApiResponse(code = 307, message = "Topic is not owned by this broker!"),
+ @ApiResponse(code = 405, message = "Pending ack handle don't use managedLedger!"),
+ @ApiResponse(code = 400, message = "Topic is not a persistent topic!"),
+ @ApiResponse(code = 409, message = "Concurrent modification")})
+ public void getPositionStatsInPendingAck(@Suspended final AsyncResponse asyncResponse,
Review Comment:
checkPositionInPendingAckState
##########
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/TransactionsImpl.java:
##########
@@ -317,10 +318,10 @@ public CompletableFuture<TransactionPendingAckInternalStats> getPendingAckIntern
path = path.queryParam("metadata", metadata);
final CompletableFuture<TransactionPendingAckInternalStats> future = new CompletableFuture<>();
asyncGetRequest(path,
- new InvocationCallback<TransactionPendingAckInternalStats>() {
+ new InvocationCallback<PositionInPendingAckStats>() {
Review Comment:
why change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#discussion_r890765839
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java:
##########
@@ -138,4 +138,33 @@ public boolean hasAckSet() {
public PositionInfo getPositionInfo() {
return PositionInfo.newBuilder().setLedgerId(ledgerId).setEntryId(entryId).build();
}
+
+ public String getAckSetAsString() {
Review Comment:
only for test, this method can delete
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java:
##########
@@ -924,6 +925,32 @@ public CompletableFuture<ManagedLedger> getStoreManageLedger() {
}
}
+ @Override
+ public PositionInPendingAckStats checkPositionInPendingAckState(PositionImpl position, Integer batchIndex) {
+ if (persistentSubscription.getCursor().getPersistentMarkDeletedPosition() != null && position.compareTo(
+ (PositionImpl) persistentSubscription.getCursor().getPersistentMarkDeletedPosition()) <= 0) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.MarkDelete);
+ } else if (individualAckPositions == null) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.NotInPendingAck);
+ }
+ MutablePair<PositionImpl, Integer> positionIntegerMutablePair = individualAckPositions.get(position);
+ if (positionIntegerMutablePair != null) {
+ if (batchIndex == null) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.PendingAck);
+ } else {
+ BitSetRecyclable bitSetRecyclable = BitSetRecyclable
+ .valueOf(positionIntegerMutablePair.left.getAckSet());
+ if (bitSetRecyclable.get(batchIndex)) {
Review Comment:
0 is in pendingAck, 1 is NotInPendingAck
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java:
##########
@@ -924,6 +925,32 @@ public CompletableFuture<ManagedLedger> getStoreManageLedger() {
}
}
+ @Override
+ public PositionInPendingAckStats checkPositionInPendingAckState(PositionImpl position, Integer batchIndex) {
+ if (persistentSubscription.getCursor().getPersistentMarkDeletedPosition() != null && position.compareTo(
+ (PositionImpl) persistentSubscription.getCursor().getPersistentMarkDeletedPosition()) <= 0) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.MarkDelete);
+ } else if (individualAckPositions == null) {
Review Comment:
should check pending ack state
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java:
##########
@@ -924,6 +925,32 @@ public CompletableFuture<ManagedLedger> getStoreManageLedger() {
}
}
+ @Override
+ public PositionInPendingAckStats checkPositionInPendingAckState(PositionImpl position, Integer batchIndex) {
+ if (persistentSubscription.getCursor().getPersistentMarkDeletedPosition() != null && position.compareTo(
+ (PositionImpl) persistentSubscription.getCursor().getPersistentMarkDeletedPosition()) <= 0) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.MarkDelete);
+ } else if (individualAckPositions == null) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.NotInPendingAck);
+ }
+ MutablePair<PositionImpl, Integer> positionIntegerMutablePair = individualAckPositions.get(position);
+ if (positionIntegerMutablePair != null) {
+ if (batchIndex == null) {
+ return new PositionInPendingAckStats(PositionInPendingAckStats.State.PendingAck);
+ } else {
+ BitSetRecyclable bitSetRecyclable = BitSetRecyclable
Review Comment:
should recycle
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -339,4 +340,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
+ @ApiOperation(value = "Get position stats in pending ack.")
+ @ApiResponses(value = {@ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Tenant or cluster or namespace or topic "
+ + "or subscription name doesn't exist"),
+ @ApiResponse(code = 503, message = "This Broker is not configured "
+ + "with transactionCoordinatorEnabled=true."),
+ @ApiResponse(code = 307, message = "Topic is not owned by this broker!"),
+ @ApiResponse(code = 405, message = "Pending ack handle don't use managedLedger!"),
+ @ApiResponse(code = 400, message = "Topic is not a persistent topic!"),
+ @ApiResponse(code = 409, message = "Concurrent modification")})
+ public void getPositionStatsInPendingAck(@Suspended final AsyncResponse asyncResponse,
+ @QueryParam("authoritative")
+ @DefaultValue("false") boolean authoritative,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @PathParam("subName") String subName,
+ @QueryParam("position") String position) {
+ try {
+ checkArgument(position != null, "Message position should not be null.");
+ checkTransactionCoordinatorEnabled();
+ validateTopicName(tenant, namespace, encodedTopic);
+ String[] args = position.split(":");
+ int length = args.length;
+ Integer batchIndex = args[2].equals("null") ? null : Integer.parseInt(args[2]);
Review Comment:
args[2] == null? check this in internalGetPositionStatsPendingAckStats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on code in PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#discussion_r905682099
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3/Transactions.java:
##########
@@ -342,4 +343,47 @@ public void scaleTransactionCoordinators(@Suspended final AsyncResponse asyncRes
resumeAsyncResponseExceptionally(asyncResponse, e);
}
}
+
+ @GET
+ @Path("/getPositionStatsInPendingAck/{tenant}/{namespace}/{topic}/{subName}")
+ @ApiOperation(value = "Get position stats in pending ack.")
+ @ApiResponses(value = {@ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Tenant or cluster or namespace or topic "
+ + "or subscription name doesn't exist"),
+ @ApiResponse(code = 503, message = "This Broker is not configured "
+ + "with transactionCoordinatorEnabled=true."),
+ @ApiResponse(code = 307, message = "Topic is not owned by this broker!"),
+ @ApiResponse(code = 405, message = "Pending ack handle don't use managedLedger!"),
+ @ApiResponse(code = 400, message = "Topic is not a persistent topic!"),
+ @ApiResponse(code = 409, message = "Concurrent modification")})
+ public void getPositionStatsInPendingAck(@Suspended final AsyncResponse asyncResponse,
+ @QueryParam("authoritative")
+ @DefaultValue("false") boolean authoritative,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @PathParam("subName") String subName,
+ @QueryParam("ledgerId") Long ledgerId,
Review Comment:
Maybe we could add the params `ledgerId` and `entryId` in the path, it's more rest style, then we don't need to validate them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou merged pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
liangyepianzhou merged PR #15682:
URL: https://github.com/apache/pulsar/pull/15682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#issuecomment-1169734717
@liangyepianzhou Please provide a correct documentation label for your PR.
Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #15682: [improve][txn] Add a admin tool to check message pending ack stats
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #15682:
URL: https://github.com/apache/pulsar/pull/15682#discussion_r904744864
##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/PositionInPendingAckStats.java:
##########
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.policies.data;
Review Comment:
move to org.apache.pulsar.common.policies.data.stats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org