You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/05 17:25:05 UTC
[GitHub] [pulsar] 315157973 opened a new pull request #11928: Fix messages in TopicPolicies will never be cleaned up
315157973 opened a new pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928
### Motivation
The current Topic Policies use compaction topic.
When a compaction topic deletes a message, the body of the message must be null.
Now, even if policies are being deleted, the body of the message is not null.
This will cause some messages in compaction topic never be cleaned up.
### Modifications
1）When deleting policies, set the message body to null
2）Add a delete API instead of modifying write API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#issuecomment-913924158
@315157973 Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702711910
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -330,9 +343,6 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
case UPDATE:
policiesCache.put(topicName, event.getPolicies());
break;
- case DELETE:
Review comment:
I added a compatibility adaptation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702649699
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -330,9 +343,6 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
case UPDATE:
policiesCache.put(topicName, event.getPolicies());
break;
- case DELETE:
Review comment:
what about old data present in existing clusters ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702553788
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -133,6 +126,21 @@ public SystemTopicBasedTopicPoliciesService(PulsarService pulsarService) {
return result;
}
+ private PulsarEvent getPulsarEvent(TopicName topicName, ActionType actionType, TopicPolicies policies) {
+ return PulsarEvent.builder()
+ .actionType(actionType)
+ .eventType(EventType.TOPIC_POLICY)
+ .topicPoliciesEvent(
+ TopicPoliciesEvent.builder()
+ .domain(topicName.getDomain().toString())
+ .tenant(topicName.getTenant())
+ .namespace(topicName.getNamespaceObject().getLocalName())
+ .topic(TopicName.get(topicName.getPartitionedTopicName()).getLocalName())
+ .policies(policies)
+ .build())
+ .build();
Review comment:
Does this code block need to be format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli merged pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
eolivelli merged pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702948239
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -331,7 +343,14 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
policiesCache.put(topicName, event.getPolicies());
break;
case DELETE:
+ // Since PR #11928, this branch is no longer needed.
+ // However, due to compatibility, it is temporarily retained here
+ // and can be deleted in the future.
policiesCache.remove(topicName);
+ SystemTopicClient<PulsarEvent> systemTopicClient = namespaceEventsSystemTopicFactory
+ .createTopicPoliciesSystemTopicClient(topicName.getNamespaceObject());
+ systemTopicClient.newWriterAsync().thenAccept(writer
+ -> writer.deleteAsync(getPulsarEvent(topicName, ActionType.DELETE, null)));
Review comment:
I have closed the Writer.
Now all Writers in SystemTopicBasedTopicPoliciesService seem to be created every time.
I think this is reasonable, after all, Topic Policies are not high-frequency operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 edited a comment on pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 edited a comment on pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#issuecomment-913937316
Hi, @Anonymitaet, I think this is not doc required, because this is a bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702888599
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -331,7 +343,14 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
policiesCache.put(topicName, event.getPolicies());
break;
case DELETE:
+ // Since PR #11928, this branch is no longer needed.
+ // However, due to compatibility, it is temporarily retained here
+ // and can be deleted in the future.
policiesCache.remove(topicName);
+ SystemTopicClient<PulsarEvent> systemTopicClient = namespaceEventsSystemTopicFactory
+ .createTopicPoliciesSystemTopicClient(topicName.getNamespaceObject());
+ systemTopicClient.newWriterAsync().thenAccept(writer
+ -> writer.deleteAsync(getPulsarEvent(topicName, ActionType.DELETE, null)));
Review comment:
The writer should be closed and for replay the historical policies, we should use one writer instead create writer for each policy data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702692670
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -330,9 +343,6 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
case UPDATE:
policiesCache.put(topicName, event.getPolicies());
break;
- case DELETE:
Review comment:
When recovering the topic policies, the reader will read messages from the earliest position, it could read the `DELETE` type Pulsar event, does it handle the `DELETE` event messages at this time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702686233
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -330,9 +343,6 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
case UPDATE:
policiesCache.put(topicName, event.getPolicies());
break;
- case DELETE:
Review comment:
Good catch, maybe we could call the delete method to publish a message with a null value in branch `DELETE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#discussion_r702687641
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -330,9 +343,6 @@ private void refreshTopicPoliciesCache(Message<PulsarEvent> msg) {
case UPDATE:
policiesCache.put(topicName, event.getPolicies());
break;
- case DELETE:
Review comment:
Indeed, for the compatibility, this code cannot be deleted now, I will add a comment here.
But Topic's existing old data seems to need to provide additional tools to delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#issuecomment-913937316
Hi, @Anonymitaet, I think this isn't doc required, because this is a bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 edited a comment on pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
gaoran10 edited a comment on pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#issuecomment-913937316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #11928: Fix messages in TopicPolicies will never be cleaned up
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #11928:
URL: https://github.com/apache/pulsar/pull/11928#issuecomment-913293439
@gaoran10 PTAL, it seems that the transaction also has this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org