You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Xiangying Meng <xi...@apache.org> on 2023/02/20 04:00:50 UTC
[DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.
Hi, Community,
We plan to add an internal class to `TransactionBufferStats` to record the
snapshot status uniformly.
As we all know, the current transaction buffer(TB) filters the messages
sent using the aborted transaction by storing the aborted ID in TB.
Then TB will periodically store these aborted txn IDs in a bookie entry in
the form of snapshots so that TB can recover faster when recovering.
But as more and more people use transactions, we found that in some extreme
cases, a bookie entry may not be able to store all aborted transaction IDs.
So in PIP196 <https://github.com/apache/pulsar/issues/16913>, we
implemented the multiple-snapshot function.
As the transaction buffer snapshot mechanism becomes increasingly complex,
the only information related to the transaction snapshot is
`lastSnapshotTimestamps`; That is not enough, we need to add more info to
record the snapshot stats.
So I suggest adding an internal class SnapshotStats to
TransactionBufferStats to record the snapshot status uniformly.
The modification could be :
```java
public class TransactionBufferStats {
...
public long lastSnapshotTimestamps;
...
}
```
```java
public class TransactionBufferStats {
...
//public long lastSnapshotTimestamps;
...
public SnapshotStats snapshotStats;
public static class SnapshotStats {
public long segmentsSize;
public long unsealedAbortTxnIDs;
public long lastSnapshotTimestamps;
}
}
```
Thanks.
Xiangying
Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.
Posted by PengHui Li <pe...@apache.org>.
Hi Xiangying,
If the change applies to users, we should start with a proposal to
provide the background, the goal and the related API changs.
Thanks,
Penghui
On Sun, Feb 26, 2023 at 11:38 PM Asaf Mesika <as...@gmail.com> wrote:
> On Fri, Feb 24, 2023 at 5:33 AM Xiangying Meng <xi...@apache.org>
> wrote:
>
> > Hi Asaf,
> > > How are those stats exposed to the user?
> > IMO, these stats should be exposed to the user.
> > The users can use it to check if the transaction snapshot is
> > running correctly.
> >
> >
> My question was "how" - prometheus, Pulsar REST API?
>
> I'm asking because this stats doesn't look very "metrics" in term of
> structure / naming
>
> > Thanks,
> > Xiangying
> >
> > On Wed, Feb 22, 2023 at 9:06 PM Asaf Mesika <as...@gmail.com>
> wrote:
> >
> > > How are those stats exposed to the user?
> > >
> > >
> > > On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng <xi...@apache.org>
> > > wrote:
> > >
> > > > Hi, Community,
> > > > We plan to add an internal class to `TransactionBufferStats` to
> record
> > > the
> > > > snapshot status uniformly.
> > > > As we all know, the current transaction buffer(TB) filters the
> messages
> > > > sent using the aborted transaction by storing the aborted ID in TB.
> > > > Then TB will periodically store these aborted txn IDs in a bookie
> entry
> > > in
> > > > the form of snapshots so that TB can recover faster when recovering.
> > > > But as more and more people use transactions, we found that in some
> > > extreme
> > > > cases, a bookie entry may not be able to store all aborted
> transaction
> > > IDs.
> > > > So in PIP196 <https://github.com/apache/pulsar/issues/16913>, we
> > > > implemented the multiple-snapshot function.
> > > > As the transaction buffer snapshot mechanism becomes increasingly
> > > complex,
> > > > the only information related to the transaction snapshot is
> > > > `lastSnapshotTimestamps`; That is not enough, we need to add more
> info
> > to
> > > > record the snapshot stats.
> > > > So I suggest adding an internal class SnapshotStats to
> > > > TransactionBufferStats to record the snapshot status uniformly.
> > > >
> > > > The modification could be :
> > > > ```java
> > > > public class TransactionBufferStats {
> > > > ...
> > > > public long lastSnapshotTimestamps;
> > > > ...
> > > > }
> > > > ```
> > > > ```java
> > > > public class TransactionBufferStats {
> > > > ...
> > > > //public long lastSnapshotTimestamps;
> > > > ...
> > > > public SnapshotStats snapshotStats;
> > > >
> > > > public static class SnapshotStats {
> > > > public long segmentsSize;
> > > >
> > > > public long unsealedAbortTxnIDs;
> > > >
> > > >
> > > > public long lastSnapshotTimestamps;
> > > > }
> > > > }
> > > >
> > > > ```
> > > > Thanks.
> > > > Xiangying
> > > >
> > >
> >
>
Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.
Posted by Asaf Mesika <as...@gmail.com>.
On Fri, Feb 24, 2023 at 5:33 AM Xiangying Meng <xi...@apache.org> wrote:
> Hi Asaf,
> > How are those stats exposed to the user?
> IMO, these stats should be exposed to the user.
> The users can use it to check if the transaction snapshot is
> running correctly.
>
>
My question was "how" - prometheus, Pulsar REST API?
I'm asking because this stats doesn't look very "metrics" in term of
structure / naming
> Thanks,
> Xiangying
>
> On Wed, Feb 22, 2023 at 9:06 PM Asaf Mesika <as...@gmail.com> wrote:
>
> > How are those stats exposed to the user?
> >
> >
> > On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng <xi...@apache.org>
> > wrote:
> >
> > > Hi, Community,
> > > We plan to add an internal class to `TransactionBufferStats` to record
> > the
> > > snapshot status uniformly.
> > > As we all know, the current transaction buffer(TB) filters the messages
> > > sent using the aborted transaction by storing the aborted ID in TB.
> > > Then TB will periodically store these aborted txn IDs in a bookie entry
> > in
> > > the form of snapshots so that TB can recover faster when recovering.
> > > But as more and more people use transactions, we found that in some
> > extreme
> > > cases, a bookie entry may not be able to store all aborted transaction
> > IDs.
> > > So in PIP196 <https://github.com/apache/pulsar/issues/16913>, we
> > > implemented the multiple-snapshot function.
> > > As the transaction buffer snapshot mechanism becomes increasingly
> > complex,
> > > the only information related to the transaction snapshot is
> > > `lastSnapshotTimestamps`; That is not enough, we need to add more info
> to
> > > record the snapshot stats.
> > > So I suggest adding an internal class SnapshotStats to
> > > TransactionBufferStats to record the snapshot status uniformly.
> > >
> > > The modification could be :
> > > ```java
> > > public class TransactionBufferStats {
> > > ...
> > > public long lastSnapshotTimestamps;
> > > ...
> > > }
> > > ```
> > > ```java
> > > public class TransactionBufferStats {
> > > ...
> > > //public long lastSnapshotTimestamps;
> > > ...
> > > public SnapshotStats snapshotStats;
> > >
> > > public static class SnapshotStats {
> > > public long segmentsSize;
> > >
> > > public long unsealedAbortTxnIDs;
> > >
> > >
> > > public long lastSnapshotTimestamps;
> > > }
> > > }
> > >
> > > ```
> > > Thanks.
> > > Xiangying
> > >
> >
>
Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.
Posted by Xiangying Meng <xi...@apache.org>.
Hi Asaf,
> How are those stats exposed to the user?
IMO, these stats should be exposed to the user.
The users can use it to check if the transaction snapshot is
running correctly.
Thanks,
Xiangying
On Wed, Feb 22, 2023 at 9:06 PM Asaf Mesika <as...@gmail.com> wrote:
> How are those stats exposed to the user?
>
>
> On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng <xi...@apache.org>
> wrote:
>
> > Hi, Community,
> > We plan to add an internal class to `TransactionBufferStats` to record
> the
> > snapshot status uniformly.
> > As we all know, the current transaction buffer(TB) filters the messages
> > sent using the aborted transaction by storing the aborted ID in TB.
> > Then TB will periodically store these aborted txn IDs in a bookie entry
> in
> > the form of snapshots so that TB can recover faster when recovering.
> > But as more and more people use transactions, we found that in some
> extreme
> > cases, a bookie entry may not be able to store all aborted transaction
> IDs.
> > So in PIP196 <https://github.com/apache/pulsar/issues/16913>, we
> > implemented the multiple-snapshot function.
> > As the transaction buffer snapshot mechanism becomes increasingly
> complex,
> > the only information related to the transaction snapshot is
> > `lastSnapshotTimestamps`; That is not enough, we need to add more info to
> > record the snapshot stats.
> > So I suggest adding an internal class SnapshotStats to
> > TransactionBufferStats to record the snapshot status uniformly.
> >
> > The modification could be :
> > ```java
> > public class TransactionBufferStats {
> > ...
> > public long lastSnapshotTimestamps;
> > ...
> > }
> > ```
> > ```java
> > public class TransactionBufferStats {
> > ...
> > //public long lastSnapshotTimestamps;
> > ...
> > public SnapshotStats snapshotStats;
> >
> > public static class SnapshotStats {
> > public long segmentsSize;
> >
> > public long unsealedAbortTxnIDs;
> >
> >
> > public long lastSnapshotTimestamps;
> > }
> > }
> >
> > ```
> > Thanks.
> > Xiangying
> >
>
Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.
Posted by Asaf Mesika <as...@gmail.com>.
How are those stats exposed to the user?
On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng <xi...@apache.org> wrote:
> Hi, Community,
> We plan to add an internal class to `TransactionBufferStats` to record the
> snapshot status uniformly.
> As we all know, the current transaction buffer(TB) filters the messages
> sent using the aborted transaction by storing the aborted ID in TB.
> Then TB will periodically store these aborted txn IDs in a bookie entry in
> the form of snapshots so that TB can recover faster when recovering.
> But as more and more people use transactions, we found that in some extreme
> cases, a bookie entry may not be able to store all aborted transaction IDs.
> So in PIP196 <https://github.com/apache/pulsar/issues/16913>, we
> implemented the multiple-snapshot function.
> As the transaction buffer snapshot mechanism becomes increasingly complex,
> the only information related to the transaction snapshot is
> `lastSnapshotTimestamps`; That is not enough, we need to add more info to
> record the snapshot stats.
> So I suggest adding an internal class SnapshotStats to
> TransactionBufferStats to record the snapshot status uniformly.
>
> The modification could be :
> ```java
> public class TransactionBufferStats {
> ...
> public long lastSnapshotTimestamps;
> ...
> }
> ```
> ```java
> public class TransactionBufferStats {
> ...
> //public long lastSnapshotTimestamps;
> ...
> public SnapshotStats snapshotStats;
>
> public static class SnapshotStats {
> public long segmentsSize;
>
> public long unsealedAbortTxnIDs;
>
>
> public long lastSnapshotTimestamps;
> }
> }
>
> ```
> Thanks.
> Xiangying
>