You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Baodi Shi <ba...@apache.org> on 2024/10/21 11:59:14 UTC
Re: [Discuss] Pulsar retention policy
hi, Thanks for bringing up this issue.
> Personally, I prefer the description of the retention policy in the
> official document, it's independent.
Me too, I'm inclined to the description of the official document.
If we want to align the implementation with the official documentation, would it be easy to refactor?
On 2024/04/15 10:31:54 太上玄元道君 wrote:
> Hi Yike,
>
> The current code implementation of the retention policy looks a little
> strange to me.
>
> The biggest problem is we have coupled the backlog quota and retention
> policy together,
> we cannot retain historical data without setting the backlog quota, say, if
> I want to retain 10GB of acknowledged messages,
> then I have to set a backlog quota.
>
> The backlog quota will block message publishing or acknowledge messages
> automatically, in some cases it's unacceptable.
>
> Personally, I prefer the description of the retention policy in the
> official document, it's independent.
>
> Thanks,
> Tao Jiuming
>
> Yike Xiao <km...@live.com> 于2024年4月13日周六 23:32写道:
>
> > Hi Jiuming,
> >
> > Thank you for bringing this up. From a Pulsar admin perspective, the
> > current retention policy implementation does not ensure that users can seek
> > back to a position within a specific size limit or have to pay extra cost
> > to achieve that. For example, to guarantee able to seek back to a position
> > 10GB earlier, users need to set the `retention policy = backlog quota +
> > 10GB`. However, the backlog quota is typically set quite large to allow for
> > significant data accumulation. Therefore, users must bear the cost of a
> > large backlog quota (e.g., 100GB) to ensure they can revert to a position
> > 10GB earlier, even if there isn't backlog in subscription.
> >
> > Regards,
> > Yike
> > ________________________________
> > From: 太上玄元道君 <da...@apache.org>
> > Sent: Thursday, April 11, 2024 18:20
> > To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> > Subject: [Discuss] Pulsar retention policy
> >
> > Hi, Pulsar community,
> >
> > I'm opening this thread to discuss the retention policy for managed
> > ledgers.
> >
> > Currently, the retention policy is defined as a time/size-based policy to
> > retain messages in the ledger, but there is a difference between the
> > official documentation and the actual code implementation.
> >
> > The official documentation states that the retention policy is to retain
> > the messages that were *acknowledged*. For example, if the retention size
> > is set to 10GB and there are 20GB of messages acknowledged, Pulsar will
> > retain 10GB and delete the rest.
> >
> > However, the actual code implementation is different. It retains the
> > messages that were *written* to the ledger, including *backlog messages*
> > and *acknowledged messages*. For instance, if there are 10GB of messages in
> > the backlog and 10GB of messages were acknowledged:
> > 1. If the retention size is set to 10GB, Pulsar will only retain the 10GB
> > of messages in the backlog, and the 10GB of messages that were acknowledged
> > will be deleted.
> > 2. If the retention size is set to 20GB, Pulsar will retain the 10GB of
> > messages in the backlog and the 10GB of messages that were acknowledged.
> > 3. If the retention size is set to 5GB, Pulsar will retain the 10GB of
> > messages in the backlog, but the 10GB of messages that were acknowledged
> > will be deleted.
> > 4. If the retention size is set to 15GB, Pulsar will retain the 10GB of
> > messages in the backlog and the 5GB of messages that were acknowledged. The
> > rest of the acknowledged messages will be deleted.
> >
> > From Pulsar open source to the present, the code implementation has never
> > changed, but the meaning of the official documentation has gradually
> > shifted. So I'm just considering which one is better: the official
> > documentation or the code implementation? Does the change in the meaning of
> > the document align more with expectations? Does it indicate that users want
> > to retain the messages that were acknowledged?
> >
> > For a long time, users have believed that the Retention Policy is for
> > retaining messages that were acknowledged. If we change the document to
> > match the code implementation, will it meet users' expectations?
> >
> > What should we do? Change the document to match the code implementation or
> > change the code implementation to match the document?
> >
> > Regards,
> > Tao Jiuming
> >
>
Re: [Discuss] Pulsar retention policy
Posted by Tao Jiuming <da...@apache.org>.
LGTM.
The pic in
https://pulsar.apache.org/docs/4.0.x/cookbooks-retention-expiry/#retention-policies
also needs change.
Thanks,
Tao Jiuming
Baodi Shi <ba...@apache.org>于2024年10月21日 周一22:08写道:
> https://github.com/apache/pulsar/issues/22473#issuecomment-2426787328
>
>
> I’ve drawn a diagram of the current backlog quota and retention
> policy. Please correct me if it's not accurate.
>
> Thanks,
> Baodi Shi
>
> Baodi Shi <ba...@apache.org> 于2024年10月21日周一 20:00写道:
> >
> > hi, Thanks for bringing up this issue.
> >
> > > Personally, I prefer the description of the retention policy in the
> > > official document, it's independent.
> >
> > Me too, I'm inclined to the description of the official document.
> >
> > If we want to align the implementation with the official documentation,
> would it be easy to refactor?
> >
> > On 2024/04/15 10:31:54 太上玄元道君 wrote:
> > > Hi Yike,
> > >
> > > The current code implementation of the retention policy looks a little
> > > strange to me.
> > >
> > > The biggest problem is we have coupled the backlog quota and retention
> > > policy together,
> > > we cannot retain historical data without setting the backlog quota,
> say, if
> > > I want to retain 10GB of acknowledged messages,
> > > then I have to set a backlog quota.
> > >
> > > The backlog quota will block message publishing or acknowledge messages
> > > automatically, in some cases it's unacceptable.
> > >
> > > Personally, I prefer the description of the retention policy in the
> > > official document, it's independent.
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > Yike Xiao <km...@live.com> 于2024年4月13日周六 23:32写道:
> > >
> > > > Hi Jiuming,
> > > >
> > > > Thank you for bringing this up. From a Pulsar admin perspective, the
> > > > current retention policy implementation does not ensure that users
> can seek
> > > > back to a position within a specific size limit or have to pay extra
> cost
> > > > to achieve that. For example, to guarantee able to seek back to a
> position
> > > > 10GB earlier, users need to set the `retention policy = backlog
> quota +
> > > > 10GB`. However, the backlog quota is typically set quite large to
> allow for
> > > > significant data accumulation. Therefore, users must bear the cost
> of a
> > > > large backlog quota (e.g., 100GB) to ensure they can revert to a
> position
> > > > 10GB earlier, even if there isn't backlog in subscription.
> > > >
> > > > Regards,
> > > > Yike
> > > > ________________________________
> > > > From: 太上玄元道君 <da...@apache.org>
> > > > Sent: Thursday, April 11, 2024 18:20
> > > > To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> > > > Subject: [Discuss] Pulsar retention policy
> > > >
> > > > Hi, Pulsar community,
> > > >
> > > > I'm opening this thread to discuss the retention policy for managed
> > > > ledgers.
> > > >
> > > > Currently, the retention policy is defined as a time/size-based
> policy to
> > > > retain messages in the ledger, but there is a difference between the
> > > > official documentation and the actual code implementation.
> > > >
> > > > The official documentation states that the retention policy is to
> retain
> > > > the messages that were *acknowledged*. For example, if the retention
> size
> > > > is set to 10GB and there are 20GB of messages acknowledged, Pulsar
> will
> > > > retain 10GB and delete the rest.
> > > >
> > > > However, the actual code implementation is different. It retains the
> > > > messages that were *written* to the ledger, including *backlog
> messages*
> > > > and *acknowledged messages*. For instance, if there are 10GB of
> messages in
> > > > the backlog and 10GB of messages were acknowledged:
> > > > 1. If the retention size is set to 10GB, Pulsar will only retain the
> 10GB
> > > > of messages in the backlog, and the 10GB of messages that were
> acknowledged
> > > > will be deleted.
> > > > 2. If the retention size is set to 20GB, Pulsar will retain the 10GB
> of
> > > > messages in the backlog and the 10GB of messages that were
> acknowledged.
> > > > 3. If the retention size is set to 5GB, Pulsar will retain the 10GB
> of
> > > > messages in the backlog, but the 10GB of messages that were
> acknowledged
> > > > will be deleted.
> > > > 4. If the retention size is set to 15GB, Pulsar will retain the 10GB
> of
> > > > messages in the backlog and the 5GB of messages that were
> acknowledged. The
> > > > rest of the acknowledged messages will be deleted.
> > > >
> > > > From Pulsar open source to the present, the code implementation has
> never
> > > > changed, but the meaning of the official documentation has gradually
> > > > shifted. So I'm just considering which one is better: the official
> > > > documentation or the code implementation? Does the change in the
> meaning of
> > > > the document align more with expectations? Does it indicate that
> users want
> > > > to retain the messages that were acknowledged?
> > > >
> > > > For a long time, users have believed that the Retention Policy is for
> > > > retaining messages that were acknowledged. If we change the document
> to
> > > > match the code implementation, will it meet users' expectations?
> > > >
> > > > What should we do? Change the document to match the code
> implementation or
> > > > change the code implementation to match the document?
> > > >
> > > > Regards,
> > > > Tao Jiuming
> > > >
> > >
>
Re: [Discuss] Pulsar retention policy
Posted by Baodi Shi <ba...@apache.org>.
https://github.com/apache/pulsar/issues/22473#issuecomment-2426787328
I’ve drawn a diagram of the current backlog quota and retention
policy. Please correct me if it's not accurate.
Thanks,
Baodi Shi
Baodi Shi <ba...@apache.org> 于2024年10月21日周一 20:00写道:
>
> hi, Thanks for bringing up this issue.
>
> > Personally, I prefer the description of the retention policy in the
> > official document, it's independent.
>
> Me too, I'm inclined to the description of the official document.
>
> If we want to align the implementation with the official documentation, would it be easy to refactor?
>
> On 2024/04/15 10:31:54 太上玄元道君 wrote:
> > Hi Yike,
> >
> > The current code implementation of the retention policy looks a little
> > strange to me.
> >
> > The biggest problem is we have coupled the backlog quota and retention
> > policy together,
> > we cannot retain historical data without setting the backlog quota, say, if
> > I want to retain 10GB of acknowledged messages,
> > then I have to set a backlog quota.
> >
> > The backlog quota will block message publishing or acknowledge messages
> > automatically, in some cases it's unacceptable.
> >
> > Personally, I prefer the description of the retention policy in the
> > official document, it's independent.
> >
> > Thanks,
> > Tao Jiuming
> >
> > Yike Xiao <km...@live.com> 于2024年4月13日周六 23:32写道:
> >
> > > Hi Jiuming,
> > >
> > > Thank you for bringing this up. From a Pulsar admin perspective, the
> > > current retention policy implementation does not ensure that users can seek
> > > back to a position within a specific size limit or have to pay extra cost
> > > to achieve that. For example, to guarantee able to seek back to a position
> > > 10GB earlier, users need to set the `retention policy = backlog quota +
> > > 10GB`. However, the backlog quota is typically set quite large to allow for
> > > significant data accumulation. Therefore, users must bear the cost of a
> > > large backlog quota (e.g., 100GB) to ensure they can revert to a position
> > > 10GB earlier, even if there isn't backlog in subscription.
> > >
> > > Regards,
> > > Yike
> > > ________________________________
> > > From: 太上玄元道君 <da...@apache.org>
> > > Sent: Thursday, April 11, 2024 18:20
> > > To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> > > Subject: [Discuss] Pulsar retention policy
> > >
> > > Hi, Pulsar community,
> > >
> > > I'm opening this thread to discuss the retention policy for managed
> > > ledgers.
> > >
> > > Currently, the retention policy is defined as a time/size-based policy to
> > > retain messages in the ledger, but there is a difference between the
> > > official documentation and the actual code implementation.
> > >
> > > The official documentation states that the retention policy is to retain
> > > the messages that were *acknowledged*. For example, if the retention size
> > > is set to 10GB and there are 20GB of messages acknowledged, Pulsar will
> > > retain 10GB and delete the rest.
> > >
> > > However, the actual code implementation is different. It retains the
> > > messages that were *written* to the ledger, including *backlog messages*
> > > and *acknowledged messages*. For instance, if there are 10GB of messages in
> > > the backlog and 10GB of messages were acknowledged:
> > > 1. If the retention size is set to 10GB, Pulsar will only retain the 10GB
> > > of messages in the backlog, and the 10GB of messages that were acknowledged
> > > will be deleted.
> > > 2. If the retention size is set to 20GB, Pulsar will retain the 10GB of
> > > messages in the backlog and the 10GB of messages that were acknowledged.
> > > 3. If the retention size is set to 5GB, Pulsar will retain the 10GB of
> > > messages in the backlog, but the 10GB of messages that were acknowledged
> > > will be deleted.
> > > 4. If the retention size is set to 15GB, Pulsar will retain the 10GB of
> > > messages in the backlog and the 5GB of messages that were acknowledged. The
> > > rest of the acknowledged messages will be deleted.
> > >
> > > From Pulsar open source to the present, the code implementation has never
> > > changed, but the meaning of the official documentation has gradually
> > > shifted. So I'm just considering which one is better: the official
> > > documentation or the code implementation? Does the change in the meaning of
> > > the document align more with expectations? Does it indicate that users want
> > > to retain the messages that were acknowledged?
> > >
> > > For a long time, users have believed that the Retention Policy is for
> > > retaining messages that were acknowledged. If we change the document to
> > > match the code implementation, will it meet users' expectations?
> > >
> > > What should we do? Change the document to match the code implementation or
> > > change the code implementation to match the document?
> > >
> > > Regards,
> > > Tao Jiuming
> > >
> >