You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/15 07:56:26 UTC
[GitHub] [pulsar] thetumbled opened a new pull request, #18936: [improve] [broker] filter system topics while shedding
thetumbled opened a new pull request, #18936:
URL: https://github.com/apache/pulsar/pull/18936
<!-- Either this PR fixes an issue, -->
Fixes #18935
### Motivation
Topics/Bundles will be unload while doing shedding, and there are some special topics that should not be unloaded for some reason. For example, if `transaction_coordinator_assign` is unloaded, the corresponding TC need to be recovered, which is time consuming.
So, we have better avoid unload these topics. And i found that such features have been implemented in the latest branch except branch-2.9.
### Modifications
fitler system topics while shedding in branch-2.9.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is already covered by existing tests, such as *(please describe tests)*.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled closed pull request #18936: [improve] [broker] filter system topics while shedding
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled closed pull request #18936: [improve] [broker] filter system topics while shedding
URL: https://github.com/apache/pulsar/pull/18936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on pull request #18936: [improve] [broker] filter system topics while shedding
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on PR #18936:
URL: https://github.com/apache/pulsar/pull/18936#issuecomment-1352815064
> In the master branch, topics/bundles in `pulsar/system` will be filter.
I think this logic was introduced by mistake by pr https://github.com/apache/pulsar/pull/15252
> and actually we can achieve distributing bundles containing `transaction_coordinator_assign` evenly and avoiding unloading these bundles while shedding at the same time.
>
> In our production clusters, we use `AvgShedder` described in #18186.
>
I will see the PIP later.
and I think the `transaction_coordinator_assign` can be shedding in any time, we could use a smoother strategy, but it doesn't prevent being shed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #18936: [improve] [broker] filter system topics while shedding
Posted by GitBox <gi...@apache.org>.
thetumbled commented on PR #18936:
URL: https://github.com/apache/pulsar/pull/18936#issuecomment-1352718553
> A system topic can be unloaded, allowing it to be load balanced. If you filter the system topic, it may lead to uneven resource allocation.
In the master branch, topics/bundles in `pulsar/system` will be filter.
`org.apache.pulsar.broker.loadbalance.LoadData#getBundleDataForLoadShedding`
<img width="634" alt="image" src="https://user-images.githubusercontent.com/52550727/207808624-94843fb9-422a-4346-a527-087c70875ed2.png">

<img width="703" alt="image" src="https://user-images.githubusercontent.com/52550727/207808970-5a496488-7465-461f-82d0-5a39a5f78cdc.png">
and actually we can achieve distributing bundles containing `transaction_coordinator_assign` evenly and avoiding unloading these bundles while shedding at the same time.
In our production clusters, we use `AvgShedder` described in https://github.com/apache/pulsar/pull/18186.
- When the cluster is initializing or the broker is restarted, bundles will be distributed randomly or based on hashing algorithm, which is similar to be a uniform distribution. So we can ensure that bundles containing `transaction_coordinator_assign` will be distributed evenly across brokers.
- When we need to do shedding, we will filter bundles containing `transaction_coordinator_assign` to avoid TC recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #18936: [improve] [broker] filter system topics while shedding
Posted by GitBox <gi...@apache.org>.
thetumbled commented on PR #18936:
URL: https://github.com/apache/pulsar/pull/18936#issuecomment-1352836432
> > In the master branch, topics/bundles in `pulsar/system` will be filter.
>
> I think this logic was introduced by mistake by pr #15252
Should i raise a PR to revert mistake in the master branch?
> and I think the transaction_coordinator_assign can be shedding in any time, we could use a smoother strategy, but it doesn't prevent being shed
I think that load balancing strategy such as `ThresholdShedder` do not work well with `transaction_coordinator_assign`, which will do many meaningless bundles unloading. and the cost of TC recovery is pretty high that there are more than 20 minutes of unavailable time in our test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18936: [improve] [broker] filter system topics while shedding
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #18936:
URL: https://github.com/apache/pulsar/pull/18936#issuecomment-1383023562
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on pull request #18936: [improve] [broker] filter system topics while shedding
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on PR #18936:
URL: https://github.com/apache/pulsar/pull/18936#issuecomment-1353008319
>
> Should i raise a PR to revert mistake in the master branch?
>
yes, I think we need a pr to revert this change
> I think that load balancing strategy such as `ThresholdShedder` do not work well with `transaction_coordinator_assign`, > which will do many meaningless bundles unloading. and the cost of TC recovery is pretty high that there are more than 20 minutes of unavailable time in our test.
I think we need to find out why the Tc recover so slowly, Is it a logic error, or need to expand the TC?
if it is a logic error, we need to fix.
later I will think about how to optimize the recovery time of TC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org