You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/04 10:20:44 UTC
[GitHub] [pulsar] jack-tristan opened a new issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction ?】
jack-tristan opened a new issue #9478:
URL: https://github.com/apache/pulsar/issues/9478
**Is your feature request related to a problem? Please describe.**
There are many data of the same stock with different prices in the stock price topic. I only need to keep the data of the latest price of the same stock. How can I delete the historical price data of the same stock? Because they take up too much storage space, I need to clean up the disk space.
**Describe the solution you'd like**
When topic compaction is enabled, you can configure whether to delete duplicate data of the same key of this topic, so as to keep only the latest data of the same key?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction enhance】
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9478:
URL: https://github.com/apache/pulsar/issues/9478#issuecomment-788959456
After discussion with @jack-tristan, the problem is enabling the retention of the topic, so the original data will not be deleted. If we want to remove the original data and just retain the compacted data, we should not specify the data retention for the topic.
If this is works for you, you can close this issue first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction enhance】
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9478:
URL: https://github.com/apache/pulsar/issues/9478#issuecomment-775640149
@jack-tristan Yes, you can use topic compaction to achieve your requirement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction enhance】
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9478:
URL: https://github.com/apache/pulsar/issues/9478#issuecomment-1058893496
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jack-tristan commented on issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction enhance】
Posted by GitBox <gi...@apache.org>.
jack-tristan commented on issue #9478:
URL: https://github.com/apache/pulsar/issues/9478#issuecomment-788548097
@codelipenghui
Pulsar topic compaction is different from Kafka compact. I need a Kafka compact to save it permanently after de duplication, so as to save space.
Pulsar topic compaction will save the original topic before compact. I need to delete the original topic and keep only the compacted topic.
How should I achieve my requirement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jack-tristan commented on issue #9478: Delete the old data of the same key in topic, so as to clean up the disk space 【topic compaction enhance】
Posted by GitBox <gi...@apache.org>.
jack-tristan commented on issue #9478:
URL: https://github.com/apache/pulsar/issues/9478#issuecomment-800013912
@codelipenghui
The experimental results are not in line with expectations
namespace policy base on pulsar 2.6.3
```
# bin/pulsar-admin namespaces get-retention test-zdh/compact-1
{
"retentionTimeInMinutes" : 0,
"retentionSizeInMB" : 0
}
# bin/pulsar-admin namespaces get-compaction-threshold test-zdh/compact-1
512
```
Experiment process:
one producer, one million messages, ten thousand keys, and one consumer's over subscription
```
# one producer,1 million messages
producer = client.newProducer(Schema.STRING)
.compressionType(CompressionType.LZ4)
.topic("persistent://test-zdh/compact-1/compact-str-v1")
.create();
String str = null;
for (int i=0; i < num; i++){
str = "num is a " + i;
String key = "key-" + i % 10000; //ten thousand keys
MessageId messageId = producer
.newMessage()
.key(key)
.value(str)
.send();
System.out.println("num: " + i);
System.out.println(key);
System.out.println(messageId.toString());
}
# one consumer
consumer = client
.newConsumer(Schema.STRING)
.topic("persistent://test-zdh/compact-1/compact-str-v1")
.subscriptionName("sub-cluster-b-test-zdh-compact-1-compact-str-v1-failover-1")
.subscriptionType(SubscriptionType.Failover) // exclusive or failover
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.readCompacted(true) //exclusive or failover persistent subscriptions
.subscribe();
```
Experimental result: The expectation is 10000, but the result is 7355
```
presto> select count(*) as num from pulsar."test-zdh/compact-1"."compact-str-v1";
num
------
7355
presto> select count(distinct(__key__)) as num from pulsar."test-zdh/compact-1"."compact-str-v1";
num
------
7355
```
```
# new consumer subscription failover-2, consume messages num is 7355
consumer = client
.newConsumer(Schema.STRING)
.topic("persistent://test-zdh/compact-1/compact-str-v1")
.subscriptionName("sub-cluster-b-test-zdh-compact-1-compact-str-v1-failover-2")
.subscriptionType(SubscriptionType.Failover) // exclusive or failover
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.readCompacted(true) //exclusive or failover persistent subscriptions
.subscribe();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org