You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/24 02:18:38 UTC
[GitHub] [pulsar] congbobo184 opened a new pull request #14830: [fix][txn]: fix transaction buffer no snapshot close recover reader
congbobo184 opened a new pull request #14830:
URL: https://github.com/apache/pulsar/pull/14830
### Motivation
now transaction buffer recover no snapshot, we don't close the reader, it will produce the problem of OOM
### Modifications
if no snapshot we should close the reader
### Verifying this change
add test for it
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a change in pull request #14830: [fix][txn]: fix transaction buffer no snapshot close recover reader
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on a change in pull request #14830:
URL: https://github.com/apache/pulsar/pull/14830#discussion_r833869917
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
##########
@@ -504,4 +506,23 @@ public void testTransactionBufferRecoverThrowBrokerMetadataException() throws Ex
txn.commit().get();
}
+
+ @Test
+ public void testTransactionBufferNoSnapshotCloseReader() throws Exception{
+ String topic = NAMESPACE1 + "/test";
+ @Cleanup
+ Producer<String> producer = pulsarClient.newProducer(Schema.STRING).producerName("testTxnTimeOut_producer")
+ .topic(topic).sendTimeout(0, TimeUnit.SECONDS).enableBatching(false).create();
+
+ admin.topics().unload(topic);
+
+ // unload success, all readers have been closed except for the compaction sub
+ producer.send("test");
+ TopicStats stats = admin.topics().getStats(NAMESPACE1 + "/" + TRANSACTION_BUFFER_SNAPSHOT);
+
+ // except for the compaction sub
+ assertEquals(stats.getSubscriptions().size(), 1);
Review comment:
because nosnapshot is a transient state, wo can't mock this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #14830: [fix][txn]: fix transaction buffer no snapshot close recover reader
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #14830:
URL: https://github.com/apache/pulsar/pull/14830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #14830: [fix][txn]: fix transaction buffer no snapshot close recover reader
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #14830:
URL: https://github.com/apache/pulsar/pull/14830#discussion_r833862028
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
##########
@@ -504,4 +506,23 @@ public void testTransactionBufferRecoverThrowBrokerMetadataException() throws Ex
txn.commit().get();
}
+
+ @Test
+ public void testTransactionBufferNoSnapshotCloseReader() throws Exception{
+ String topic = NAMESPACE1 + "/test";
+ @Cleanup
+ Producer<String> producer = pulsarClient.newProducer(Schema.STRING).producerName("testTxnTimeOut_producer")
+ .topic(topic).sendTimeout(0, TimeUnit.SECONDS).enableBatching(false).create();
+
+ admin.topics().unload(topic);
+
+ // unload success, all readers have been closed except for the compaction sub
+ producer.send("test");
+ TopicStats stats = admin.topics().getStats(NAMESPACE1 + "/" + TRANSACTION_BUFFER_SNAPSHOT);
+
+ // except for the compaction sub
+ assertEquals(stats.getSubscriptions().size(), 1);
Review comment:
Do you need to check if the subscriptions should > 1 before and then be clean up later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #14830: [fix][txn]: fix transaction buffer no snapshot close recover reader
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #14830:
URL: https://github.com/apache/pulsar/pull/14830#discussion_r833872565
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
##########
@@ -575,6 +575,10 @@ public void run() {
}
}
if (!hasSnapshot) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer reader close error!", topic.getName(), e);
+ return null;
+ });
Review comment:
Maybe we could add a method like `closeCursor` to close the `Reader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org