You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Hang Chen <ch...@apache.org> on 2024/05/04 15:12:08 UTC
[DISCUSS] PIP-348: Trigger offload on topic load stage
Hi guys,
I pushed a new proposal to improve the tiered storage offload
efficiency. Please help take a look, thanks.
PIP: https://github.com/apache/pulsar/pull/22650
Thanks,
Hang
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Hang Chen <ch...@apache.org>.
Hi guys,
If there are no objections, I will send out the vote.
Best,
Hang
Hang Chen <ch...@apache.org> 于2024年5月6日周一 22:23写道：
>
> >I see, but when a broker loads a bundle with many topics, you trigger
> immediately offloading for them, even if they don't have much traffic.
> This is kind of a "bad surprise" when you update a cluster to the newer
> version and you don't expect load to spike unexpectedly.
> My concerns is only about the upgrade path for users coming from older
> versions
>
> I see. That makes sense. Let me add a flag to control this feature.
>
> Thanks,
> Hang
>
> Enrico Olivelli <eo...@gmail.com> 于2024年5月6日周一 15:21写道：
> >
> > Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <ch...@apache.org> ha
> > scritto:
> >
> > > Hi Enrico,
> > > I agree with Jiuming, we do not need a flag to control this
> > > behavior. We will call `maybeOffloadInBackground` method in the
> > > managedledger init stage. This method only checks the offload policies
> > > to decide if we need to trigger offload. If the offload conditions are
> > > met, use the scheduled executor to trigger offload. The whole check
> > > process has no blocking calls and won't impact the managedledger init.
> > >
> > > https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
> > >
> > >
> >
> > I see, but when a broker loads a bundle with many topics, you trigger
> > immediately offloading for them, even if they don't have much traffic.
> > This is kind of a "bad surprise" when you update a cluster to the newer
> > version and you don't expect load to spike unexpectedly.
> > My concerns is only about the upgrade path for users coming from older
> > versions
> >
> > Maybe it is not a big deal ?
> >
> > Enrico
> >
> >
> > >
> > > Thanks,
> > > Hang
> > >
> > > 太上玄元道君 <da...@apache.org> 于2024年5月6日周一 00:31写道：
> > > >
> > > > hi Enrico,
> > > >
> > > > I still think it doesn't matter
> > > >
> > > > 1. Offload a topic or not, configured by
> > > >
> > > `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> > > > even though we may trigger a bunch of stuff, it's expected behavior
> > > >
> > > > 2. After triggered to offload a topic, the offload task will be added to
> > > an
> > > > Executor, currently, we have 2 threads to execute offload with sync mode,
> > > > so, even though there are thousands of topics, only 2 tasks can be
> > > executed
> > > > at the same time, other tasks wait.
> > > >
> > > > 3. If you are worrying about system overwhelmed, there is a PR to
> > > introduce
> > > > offload throttling(https://github.com/apache/pulsar/pull/22385),
> > > > it is for the purpose of protect the broker, it has been pending in
> > > review
> > > > for a loooong time, I'm very appreciate it that if you can help review
> > > >
> > > > 4. There are toooo many flags in pulsar(~500), I think we should be
> > > > cautious when introducing new FLAGs
> > > >
> > > > Thanks,
> > > > Tao Jiuming
> > > >
> > > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日13:44写道：
> > > >
> > > > > daojun,
> > > > >
> > > > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
> > > > > scritto:
> > > > >
> > > > > > it doesn't matter, we don't need to wait until offload finished when
> > > > > > loading topic, since offloading may take a long time.
> > > > > > it just trigger offloading, and we don't need to handle the result.
> > > > > >
> > > > >
> > > > > We don't have to wait for the results, but we still trigger a bunch of
> > > > > stuff and if the broker has thousand of topics, there may be an herd
> > > effect
> > > > > as this is a behavior change, I think it is safest to have a flag to
> > > > > disable this new behavior,
> > > > > otherwise upgrading an existing big cluster may lead to unexpected
> > > results,
> > > > > like overwhelmed systems
> > > > >
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> > > > > >
> > > > > > > Overall I think it is a good idea.
> > > > > > >
> > > > > > > What about adding a flag to enable this behavior?
> > > > > > >
> > > > > > > In case there are problems with the offloader the topic won't be
> > > able
> > > > > to
> > > > > > > load.
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha
> > > scritto:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > > I pushed a new proposal to improve the tiered storage
> > > offload
> > > > > > > > efficiency. Please help take a look, thanks.
> > > > > > > >
> > > > > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Hang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Hang Chen <ch...@apache.org>.
>I see, but when a broker loads a bundle with many topics, you trigger
immediately offloading for them, even if they don't have much traffic.
This is kind of a "bad surprise" when you update a cluster to the newer
version and you don't expect load to spike unexpectedly.
My concerns is only about the upgrade path for users coming from older
versions
I see. That makes sense. Let me add a flag to control this feature.
Thanks,
Hang
Enrico Olivelli <eo...@gmail.com> 于2024年5月6日周一 15:21写道：
>
> Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <ch...@apache.org> ha
> scritto:
>
> > Hi Enrico,
> > I agree with Jiuming, we do not need a flag to control this
> > behavior. We will call `maybeOffloadInBackground` method in the
> > managedledger init stage. This method only checks the offload policies
> > to decide if we need to trigger offload. If the offload conditions are
> > met, use the scheduled executor to trigger offload. The whole check
> > process has no blocking calls and won't impact the managedledger init.
> >
> > https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
> >
> >
>
> I see, but when a broker loads a bundle with many topics, you trigger
> immediately offloading for them, even if they don't have much traffic.
> This is kind of a "bad surprise" when you update a cluster to the newer
> version and you don't expect load to spike unexpectedly.
> My concerns is only about the upgrade path for users coming from older
> versions
>
> Maybe it is not a big deal ?
>
> Enrico
>
>
> >
> > Thanks,
> > Hang
> >
> > 太上玄元道君 <da...@apache.org> 于2024年5月6日周一 00:31写道：
> > >
> > > hi Enrico,
> > >
> > > I still think it doesn't matter
> > >
> > > 1. Offload a topic or not, configured by
> > >
> > `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> > > even though we may trigger a bunch of stuff, it's expected behavior
> > >
> > > 2. After triggered to offload a topic, the offload task will be added to
> > an
> > > Executor, currently, we have 2 threads to execute offload with sync mode,
> > > so, even though there are thousands of topics, only 2 tasks can be
> > executed
> > > at the same time, other tasks wait.
> > >
> > > 3. If you are worrying about system overwhelmed, there is a PR to
> > introduce
> > > offload throttling(https://github.com/apache/pulsar/pull/22385),
> > > it is for the purpose of protect the broker, it has been pending in
> > review
> > > for a loooong time, I'm very appreciate it that if you can help review
> > >
> > > 4. There are toooo many flags in pulsar(~500), I think we should be
> > > cautious when introducing new FLAGs
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日13:44写道：
> > >
> > > > daojun,
> > > >
> > > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
> > > > scritto:
> > > >
> > > > > it doesn't matter, we don't need to wait until offload finished when
> > > > > loading topic, since offloading may take a long time.
> > > > > it just trigger offloading, and we don't need to handle the result.
> > > > >
> > > >
> > > > We don't have to wait for the results, but we still trigger a bunch of
> > > > stuff and if the broker has thousand of topics, there may be an herd
> > effect
> > > > as this is a behavior change, I think it is safest to have a flag to
> > > > disable this new behavior,
> > > > otherwise upgrading an existing big cluster may lead to unexpected
> > results,
> > > > like overwhelmed systems
> > > >
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > >
> > > > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> > > > >
> > > > > > Overall I think it is a good idea.
> > > > > >
> > > > > > What about adding a flag to enable this behavior?
> > > > > >
> > > > > > In case there are problems with the offloader the topic won't be
> > able
> > > > to
> > > > > > load.
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha
> > scritto:
> > > > > >
> > > > > > > Hi guys,
> > > > > > > I pushed a new proposal to improve the tiered storage
> > offload
> > > > > > > efficiency. Please help take a look, thanks.
> > > > > > >
> > > > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Hang
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <ch...@apache.org> ha
scritto:
> Hi Enrico,
> I agree with Jiuming, we do not need a flag to control this
> behavior. We will call `maybeOffloadInBackground` method in the
> managedledger init stage. This method only checks the offload policies
> to decide if we need to trigger offload. If the offload conditions are
> met, use the scheduled executor to trigger offload. The whole check
> process has no blocking calls and won't impact the managedledger init.
>
> https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
>
>
I see, but when a broker loads a bundle with many topics, you trigger
immediately offloading for them, even if they don't have much traffic.
This is kind of a "bad surprise" when you update a cluster to the newer
version and you don't expect load to spike unexpectedly.
My concerns is only about the upgrade path for users coming from older
versions
Maybe it is not a big deal ?
Enrico
>
> Thanks,
> Hang
>
> 太上玄元道君 <da...@apache.org> 于2024年5月6日周一 00:31写道：
> >
> > hi Enrico,
> >
> > I still think it doesn't matter
> >
> > 1. Offload a topic or not, configured by
> >
> `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> > even though we may trigger a bunch of stuff, it's expected behavior
> >
> > 2. After triggered to offload a topic, the offload task will be added to
> an
> > Executor, currently, we have 2 threads to execute offload with sync mode,
> > so, even though there are thousands of topics, only 2 tasks can be
> executed
> > at the same time, other tasks wait.
> >
> > 3. If you are worrying about system overwhelmed, there is a PR to
> introduce
> > offload throttling(https://github.com/apache/pulsar/pull/22385),
> > it is for the purpose of protect the broker, it has been pending in
> review
> > for a loooong time, I'm very appreciate it that if you can help review
> >
> > 4. There are toooo many flags in pulsar(~500), I think we should be
> > cautious when introducing new FLAGs
> >
> > Thanks,
> > Tao Jiuming
> >
> > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日13:44写道：
> >
> > > daojun,
> > >
> > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
> > > scritto:
> > >
> > > > it doesn't matter, we don't need to wait until offload finished when
> > > > loading topic, since offloading may take a long time.
> > > > it just trigger offloading, and we don't need to handle the result.
> > > >
> > >
> > > We don't have to wait for the results, but we still trigger a bunch of
> > > stuff and if the broker has thousand of topics, there may be an herd
> effect
> > > as this is a behavior change, I think it is safest to have a flag to
> > > disable this new behavior,
> > > otherwise upgrading an existing big cluster may lead to unexpected
> results,
> > > like overwhelmed systems
> > >
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> > > >
> > > > > Overall I think it is a good idea.
> > > > >
> > > > > What about adding a flag to enable this behavior?
> > > > >
> > > > > In case there are problems with the offloader the topic won't be
> able
> > > to
> > > > > load.
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha
> scritto:
> > > > >
> > > > > > Hi guys,
> > > > > > I pushed a new proposal to improve the tiered storage
> offload
> > > > > > efficiency. Please help take a look, thanks.
> > > > > >
> > > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > > >
> > > > > > Thanks,
> > > > > > Hang
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Hang Chen <ch...@apache.org>.
Hi Enrico,
I agree with Jiuming, we do not need a flag to control this
behavior. We will call `maybeOffloadInBackground` method in the
managedledger init stage. This method only checks the offload policies
to decide if we need to trigger offload. If the offload conditions are
met, use the scheduled executor to trigger offload. The whole check
process has no blocking calls and won't impact the managedledger init.
https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
Thanks,
Hang
太上玄元道君 <da...@apache.org> 于2024年5月6日周一 00:31写道：
>
> hi Enrico,
>
> I still think it doesn't matter
>
> 1. Offload a topic or not, configured by
> `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> even though we may trigger a bunch of stuff, it's expected behavior
>
> 2. After triggered to offload a topic, the offload task will be added to an
> Executor, currently, we have 2 threads to execute offload with sync mode,
> so, even though there are thousands of topics, only 2 tasks can be executed
> at the same time, other tasks wait.
>
> 3. If you are worrying about system overwhelmed, there is a PR to introduce
> offload throttling(https://github.com/apache/pulsar/pull/22385),
> it is for the purpose of protect the broker, it has been pending in review
> for a loooong time, I'm very appreciate it that if you can help review
>
> 4. There are toooo many flags in pulsar(~500), I think we should be
> cautious when introducing new FLAGs
>
> Thanks,
> Tao Jiuming
>
> Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日13:44写道：
>
> > daojun,
> >
> > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
> > scritto:
> >
> > > it doesn't matter, we don't need to wait until offload finished when
> > > loading topic, since offloading may take a long time.
> > > it just trigger offloading, and we don't need to handle the result.
> > >
> >
> > We don't have to wait for the results, but we still trigger a bunch of
> > stuff and if the broker has thousand of topics, there may be an herd effect
> > as this is a behavior change, I think it is safest to have a flag to
> > disable this new behavior,
> > otherwise upgrading an existing big cluster may lead to unexpected results,
> > like overwhelmed systems
> >
> >
> > Enrico
> >
> >
> >
> > >
> > > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> > >
> > > > Overall I think it is a good idea.
> > > >
> > > > What about adding a flag to enable this behavior?
> > > >
> > > > In case there are problems with the offloader the topic won't be able
> > to
> > > > load.
> > > >
> > > > Enrico
> > > >
> > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha scritto:
> > > >
> > > > > Hi guys,
> > > > > I pushed a new proposal to improve the tiered storage offload
> > > > > efficiency. Please help take a look, thanks.
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > >
> > > > > Thanks,
> > > > > Hang
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by 太上玄元道君 <da...@apache.org>.
hi Enrico,
I still think it doesn't matter
1. Offload a topic or not, configured by
`managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
even though we may trigger a bunch of stuff, it's expected behavior
2. After triggered to offload a topic, the offload task will be added to an
Executor, currently, we have 2 threads to execute offload with sync mode,
so, even though there are thousands of topics, only 2 tasks can be executed
at the same time, other tasks wait.
3. If you are worrying about system overwhelmed, there is a PR to introduce
offload throttling(https://github.com/apache/pulsar/pull/22385),
it is for the purpose of protect the broker, it has been pending in review
for a loooong time, I'm very appreciate it that if you can help review
4. There are toooo many flags in pulsar(~500), I think we should be
cautious when introducing new FLAGs
Thanks,
Tao Jiuming
Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日13:44写道：
> daojun,
>
> Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
> scritto:
>
> > it doesn't matter, we don't need to wait until offload finished when
> > loading topic, since offloading may take a long time.
> > it just trigger offloading, and we don't need to handle the result.
> >
>
> We don't have to wait for the results, but we still trigger a bunch of
> stuff and if the broker has thousand of topics, there may be an herd effect
> as this is a behavior change, I think it is safest to have a flag to
> disable this new behavior,
> otherwise upgrading an existing big cluster may lead to unexpected results,
> like overwhelmed systems
>
>
> Enrico
>
>
>
> >
> > Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> >
> > > Overall I think it is a good idea.
> > >
> > > What about adding a flag to enable this behavior?
> > >
> > > In case there are problems with the offloader the topic won't be able
> to
> > > load.
> > >
> > > Enrico
> > >
> > > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha scritto:
> > >
> > > > Hi guys,
> > > > I pushed a new proposal to improve the tiered storage offload
> > > > efficiency. Please help take a look, thanks.
> > > >
> > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > >
> >
>
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Enrico Olivelli <eo...@gmail.com>.
daojun,
Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <da...@apache.org> ha
scritto:
> it doesn't matter, we don't need to wait until offload finished when
> loading topic, since offloading may take a long time.
> it just trigger offloading, and we don't need to handle the result.
>
We don't have to wait for the results, but we still trigger a bunch of
stuff and if the broker has thousand of topics, there may be an herd effect
as this is a behavior change, I think it is safest to have a flag to
disable this new behavior,
otherwise upgrading an existing big cluster may lead to unexpected results,
like overwhelmed systems
Enrico
>
> Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
>
> > Overall I think it is a good idea.
> >
> > What about adding a flag to enable this behavior?
> >
> > In case there are problems with the offloader the topic won't be able to
> > load.
> >
> > Enrico
> >
> > Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha scritto:
> >
> > > Hi guys,
> > > I pushed a new proposal to improve the tiered storage offload
> > > efficiency. Please help take a look, thanks.
> > >
> > > PIP: https://github.com/apache/pulsar/pull/22650
> > >
> > > Thanks,
> > > Hang
> > >
> >
>
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by 太上玄元道君 <da...@apache.org>.
it doesn't matter, we don't need to wait until offload finished when
loading topic, since offloading may take a long time.
it just trigger offloading, and we don't need to handle the result.
Enrico Olivelli <eo...@gmail.com>于2024年5月5日 周日02:41写道：
> Overall I think it is a good idea.
>
> What about adding a flag to enable this behavior?
>
> In case there are problems with the offloader the topic won't be able to
> load.
>
> Enrico
>
> Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha scritto:
>
> > Hi guys,
> > I pushed a new proposal to improve the tiered storage offload
> > efficiency. Please help take a look, thanks.
> >
> > PIP: https://github.com/apache/pulsar/pull/22650
> >
> > Thanks,
> > Hang
> >
>
Re: [DISCUSS] PIP-348: Trigger offload on topic load stage
Posted by Enrico Olivelli <eo...@gmail.com>.
Overall I think it is a good idea.
What about adding a flag to enable this behavior?
In case there are problems with the offloader the topic won't be able to
load.
Enrico
Il Sab 4 Mag 2024, 17:12 Hang Chen <ch...@apache.org> ha scritto:
> Hi guys,
> I pushed a new proposal to improve the tiered storage offload
> efficiency. Please help take a look, thanks.
>
> PIP: https://github.com/apache/pulsar/pull/22650
>
> Thanks,
> Hang
>