You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/20 03:16:02 UTC
[GitHub] [pulsar] 315157973 opened a new issue, #18128: PIP-XYZ: Support full scan and trim ledger
315157973 opened a new issue, #18128:
URL: https://github.com/apache/pulsar/issues/18128
### Motivation
Broker uses the `Trimledgers` thread to clean up outdated ledgers. During clearing, each Broker traverses the topic metadata in memory to find the ledger that reach the retention or TTL threshold.
However, there are some problems with this approach. When a topic has no producer and consumer, Broker deletes the metadata of topic from memory. As a result, ledgers of these topics can never be deleted.
Therefore, we need a way to scan and clean all outdated ledgers .
### Goal
The full scan will cause a large number of requests to the ZooKeeper.
Therefore, the existing cleanup mode will be retained and a full scan mode will be added.
### API Changes
1. Add a new scheduling thread pool
2. Add the following configuration item:
# Full scan interval. This parameter is enabled only when the value > 0.
fullScanTrimLedgerInterval=0
# Maximum number of ZooKeeper requests per second during scanning
fullScanMaximumZooKeeperRequestsPerSecond=200
### Implementation
1. Only the Leader Broker performs full scan.
2. Leader Broker traverse `managedLedger` in each namespace from ZK. Since Ledger metadata contains the creation time. If the creation time is greater than the retention time + TTL time, Ledger should be deleted.
Only the metadata of Ledger is parsed instead of loading all topics to the memory.
The zk request frequency is limited using semaphore.
3. When a topic that meets the conditions, the leader broker loads the topic and invokes its `TrimLedger` method. After cleaning is done, the leader closes the topic to release memory.
### Alternatives
_No response_
### Anything else?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1288748909
fullScanMaximumZooKeeperRequestsPerSecond=200 -> fullScanMaximumMetadataRequestsPerSecond=200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304708181
> > 2. // Full scan interval. This parameter is enabled only when the value > 0.
> > fullScanTrimLedgerInterval=0
>
> It seems that this kind of cleanup may have potential impact on the metadata store, I am not sure a fixed interval is the best option here. For the best of service stability, can we provide a fix time range that this clean up work could be triggered. For example, this can only be triggered at midnight, or during 00:00 - 05:00 everyday.
>
> WDYT? @315157973
Why not create a new PIP and make all schedule tasks support expressions like crontab?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1288344161
> Please mention that this impact offloaded data.
Offload is not affected because the current `TrimLedger` mode is reused for cleaning. We only load the topics that have not been loaded to the memory for a long time on this basis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1305073899
> I am not sure exactly how the zombie topics come from, but from this description, the number may be accumulated for a long time with old clusters. Once user upgraded to a version with this feature, they may need to cleanup a large number of these topics at the first time.
>
> From what I see, the total time required to do the cleanup work does not matter that much, maybe even single thread is good enough. But we need to minimize the impact on the server stability.
The scene where the zombie topic appears:
1) Some businesses try out some topics, but they haven't used this topic since then
2) Some topics that have data but have not been used since then, and they have not been deleted
The current `TrimLedger` is to concurrently traverse the Topics in memory and clean up all Topics. Do you mean that this parallel cleanup will also affect the stability of the Broker, so it need a single thread?
I will modify current `TrimLedger` and add a single-thread thread pool to queue these tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1289889322
> Great PIP.
>
> > 2. The metadata request frequency is limited using semaphore.
>
> Sounds like we are limiting request concurrency, not QPS from the definition of `fullScanMaximumMetadataRequestsPerSecond`.
>
> > 3. When a topic that meets the conditions, the leader broker loads the topic
>
> Shouldn't be the bundle owner loading the topic?
> Sounds like we are limiting request concurrency, not QPS from the definition of `fullScanMaximumMetadataRequestsPerSecond`.
change to `fullScanMaximumMetadataConcurrencyPerSecond`
> Shouldn't be the bundle owner loading the topic?
In this way, all nodes need to read the topic data once and check whether the topic data belongs to them based on the bundle. I haven't figured out any good way for the leader to proactively notify other brokers to load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304428164
> 2\. // Full scan interval. This parameter is enabled only when the value > 0.
> fullScanTrimLedgerInterval=0
It seems that this kind of cleanup may have potential impact on the metadata store, I am not sure a fixed interval is the best option here. For the best of service stability, can we provide a fix time range that this clean up work could be triggered. For example, this can only be triggered at midnight, or during 00:00 - 05:00 everyday.
WDYT?
@315157973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1302904213
> @315157973 Thanks for your proposal. I have a few concerns.
>
> 1. Should we check whether the managedLedger has been loaded into brokers before checking the retention policy? Otherwise, the leader needs to check all the managedLedgers
> 2. If one topic has 10000 ledgers and only one ledger needs to be deleted, do we need to load and trigger trim?
> 3. The topic retention policy has retention time and retention size, we also need to take retention size into consideration because some topics may only configure retention size.
> 4. We have broker-level, namespace-level, and topic-level retention policies, how do we get each topic's retention policy before checking the ledger's retention in meta store
> 5. If the user enabled tiered storage, we also need to take offloaded ledger's retention into consideration
> 6. Due to the bundle being the basic unit of loading, If the leader loads topics and trims them one by one, we may load a lot of bundles into the leader and may bring more load to the load balance.
1. Broker judges whether it needs to be loaded according to the creation time and size in the metadata.
If we only check the creation time, we don't need to scan all `ManagedLedgers` because they are ordered. Because we also check the size, we need to scan all.
2. Yes
3. I'll add it as a judgment item. A side effect is that all `ManagedLedgers` need to be scanned
4. When scanning, only the Broker-level Policy is used as a rough filter. The `trimLedger` method will load the full policy for judgment.
5. Since the `trimLedger` method of topic is reused, this scenario has been covered
6. I have changed to expose an admin API of `trimLedger`, Leader is only responsible for triggering, and the corresponding Broker is responsible for loading Topic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1294457977
one question.
leader load the topic, may cause the normal topic become fence. may need to consider how to not affect the normal topic behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304724483
> Why not create a new PIP and make all schedule tasks support expressions like crontab?
Make sense, I had a quick look at the schedule tasks in broker. It seems this can apply to most Inactivity Monitor tasks in `org.apache.pulsar.broker.service.BrokerService#startInactivityMonitor`.
I believe it would be an interesting PIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1286535720
Please mention that this impact offloaded data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1305020907
> When a topic has no producer and consumer, Broker deletes the metadata of topic from memory. As a result, ledgers of these topics can never be deleted.
I am not sure exactly how the zombie topics come from, but from this description, the number may be accumulated for a long time with old clusters. Once user upgraded to a version with this feature, they may need to cleanup a large number of these topics at the first time.
From what I see, the total time required to do the cleanup work does not matter that much, maybe even single thread is good enough. But we need to minimize the impact on the server stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1290225367
> Yes, it would require much more work on the coordination. But we have to consider the case when consumer/producer connects during the cleanup.
We only need to check whether producers and consumers exist to determine whether to close topic. In addition, the number of zombie topics is not large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304725526
> * // Maximum number of Metadata requests per second during scanning
> fullScanMaximumMetadataConcurrencyPerSecond=200
> * Add a `TrimLedger` admin API.
IIUC, the `TrimLedger` admin API should be synchronized, right? I mean that it should send response to leader broker after the cleaning is finished, so that the leader broker can release the semaphore for the concurrency of `fullScanMaximumMetadataConcurrencyPerSecond`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304741531
>
Garbage collection will take a long time and synchronization doesn't seem feasible right?
I don't think there are many real zombie topics. The main overhead will be that the leader reads ZK and parses the metadata, and there are not many really eligible topics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1341869099
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1368181773
> ### Motivation
> Broker uses the `Trimledgers` thread to clean up outdated ledgers. During cleaning, each Broker traverses the topic metadata in memory to find the ledger that reach the retention or TTL threshold. However, there are some problems with this approach. When a topic has no producer and consumer, Broker deletes the metadata of topic from memory. As a result, ledgers of these topics can never be deleted. Therefore, we need a way to scan and clean all outdated ledgers .
>
> ### Goal
> The full scan will cause a large number of requests to the ZooKeeper. Therefore, the existing cleanup mode will be retained and a full scan mode will be added.
>
> ### API Changes
> 1. Add a new scheduling thread pool
> 2. Add the following configuration item:
> // Full scan interval. This parameter is enabled only when the value > 0.
> fullScanTrimLedgerInterval=0
> // Maximum number of Metadata requests per second during scanning
> fullScanMaximumMetadataConcurrencyPerSecond=200
> 3. Add a `TrimLedger` admin API.
>
> ### Implementation
> 1. Only the Leader Broker performs full scan.
> 2. Leader Broker traverse `managedLedger` in each namespace from meta store . Since Ledger metadata contains the creation time. If the creation time is greater than the retention time + TTL time or size, Ledger should be deleted.
> Only the metadata of Ledger is parsed instead of loading all topics to the memory.
> The metadata request frequency is limited using semaphore.
> 3. When a topic that meets the conditions, the Leader triggers the `TrimLedger` admin API. Since the admin API will verify the attribution of the topic and redirect it, the topic will be loaded by the corresponding Broker.
> After cleaning is done, the corresponding Broker closes the topic to release memory(Before closing, it will check whether the topic has producers and consumers, if not, close it).
>
> ### Alternatives
> _No response_
>
> ### Anything else?
> _No response_
Nice work, we have encountered the same problem, detailed information can refer to: https://github.com/apache/pulsar/issues/19077
The current Ledger cleaning mechanism of Pulsar Broker is indeed not rigorous enough, which will cause a lot of dirty data to be missed in Bookie's EntryLog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1290094021
> In this way, all nodes need to read the topic data once and check whether the topic data belongs to them based on the bundle. I haven't figured out any good way for the leader to proactively notify other brokers to load.
Yes, it would require much more work on the coordination. But we have to consider the case when consumer/producer connects during the cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #18128: [PIP] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1289875186
Great PIP.
> 2\. The metadata request frequency is limited using semaphore.
Sounds like we are limiting request concurrency, not QPS from the definition of `fullScanMaximumMetadataRequestsPerSecond`.
> 3\. When a topic that meets the conditions, the leader broker loads the topic
Shouldn't be the bundle owner loading the topic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1300319803
@315157973 Thanks for your proposal. I have a few concerns.
1. Should we check whether the managedLedger has been loaded into brokers before checking the retention policy? Otherwise, the leader needs to check all the managedLedgers
2. If one topic has 10000 ledgers and only one ledger needs to be deleted, do we need to load and trigger trim?
3. The topic retention policy has retention time and retention size, we also need to take retention size into consideration because some topics may only configure retention size.
4. We have broker-level, namespace-level, and topic-level retention policies, how do we get each topic's retention policy before checking the ledger's retention in meta store
5. If the user enabled tiered storage, we also need to take offloaded ledger's retention into consideration
6. Due to the bundle being the basic unit of loading, If the leader loads topics and trims them one by one, we may load a lot of bundles into the leader and may bring more load to the load balance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1294542184
> one question. leader load the topic, may cause the normal topic become fence. may need to consider how to not affect the normal topic behavior.
Normal topics will not be loaded. Because metadata of normal topics will not meet the conditions of full scan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1304741561
> > * // Maximum number of Metadata requests per second during scanning
> > fullScanMaximumMetadataConcurrencyPerSecond=200
> > * Add a `TrimLedger` admin API.
>
> IIUC, the `TrimLedger` admin API should be synchronized, right? I mean that it should send response to leader broker after the cleaning is finished, so that the leader broker can release the semaphore for the concurrency of `fullScanMaximumMetadataConcurrencyPerSecond`.
Garbage collection will take a long time and synchronization doesn't seem feasible right?
I don't think there are many real zombie topics. The main overhead will be that the leader reads ZK and parses the metadata, and there are not many really eligible topics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18128: [PIP-219] Support full scan and trim ledger
Posted by github-actions.
github-actions[bot] commented on issue #18128:
URL: https://github.com/apache/pulsar/issues/18128#issuecomment-1411346044
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org