You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Tao Jiuming <da...@apache.org> on 2025/02/11 06:37:59 UTC
[VOTE] PIP-404： Introduce per-ledger properties
Hi all,
I open PIP-404 to introduce per ledger properties, which will add a
`properties` field to `LedgerInfo` to store extra properties for every
`Ledger`.
# Background knowledge
As we don't have a secondary index on the Bookkeeper, so we can't query
entries by message metadata efficiently.
The `ManagedCursor` provides a method `asyncFindNewestMatching` to find the
newest entry that matches the given
predicate by binary search(See `OpFindNewest.java`).
In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
timestamp` by calculating
the range of ledgers that may contain the target timestamp by
`LedgerInfo#timestamp` and we don't need to scan all
ledgers.
However, when we enabled `AppendIndexMetadataInterceptor` and we want to
query entries by `BrokerEntryMetadata#index`,
there is no more efficient way,
we have to scan all ledgers by binary search to find the target entry.
# Motivation
Introduce per-ledger properties and we can store the extra per-ledger
properties in the `LedgerInfo`,
so we can query entries by `incremental index` more efficiently, say,
`BrokerEntryMetadata#index`.
PR link: https://github.com/apache/pulsar/pull/23837/files
Please help review the PIP and vote.
Thanks,
Tao Jiuming(dao-jun)
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Yunze Xu <xy...@apache.org>.
+1 (binding)
Thanks,
Yunze
On Tue, Feb 18, 2025 at 6:00 PM Lari Hotari <lh...@apache.org> wrote:
>
> +1 (binding)
>
> -Lari
>
> On 2025/02/11 06:37:59 Tao Jiuming wrote:
> > Hi all,
> >
> > I open PIP-404 to introduce per ledger properties, which will add a
> > `properties` field to `LedgerInfo` to store extra properties for every
> > `Ledger`.
> >
> > # Background knowledge
> > As we don't have a secondary index on the Bookkeeper, so we can't query
> > entries by message metadata efficiently.
> > The `ManagedCursor` provides a method `asyncFindNewestMatching` to find the
> > newest entry that matches the given
> > predicate by binary search(See `OpFindNewest.java`).
> > In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> > timestamp` by calculating
> > the range of ledgers that may contain the target timestamp by
> > `LedgerInfo#timestamp` and we don't need to scan all
> > ledgers.
> > However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> > query entries by `BrokerEntryMetadata#index`,
> > there is no more efficient way,
> > we have to scan all ledgers by binary search to find the target entry.
> >
> > # Motivation
> > Introduce per-ledger properties and we can store the extra per-ledger
> > properties in the `LedgerInfo`,
> > so we can query entries by `incremental index` more efficiently, say,
> > `BrokerEntryMetadata#index`.
> >
> > PR link: https://github.com/apache/pulsar/pull/23837/files
> >
> > Please help review the PIP and vote.
> >
> > Thanks,
> > Tao Jiuming(dao-jun)
> >
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Lari Hotari <lh...@apache.org>.
+1 (binding)
-Lari
On 2025/02/11 06:37:59 Tao Jiuming wrote:
> Hi all,
>
> I open PIP-404 to introduce per ledger properties, which will add a
> `properties` field to `LedgerInfo` to store extra properties for every
> `Ledger`.
>
> # Background knowledge
> As we don't have a secondary index on the Bookkeeper, so we can't query
> entries by message metadata efficiently.
> The `ManagedCursor` provides a method `asyncFindNewestMatching` to find the
> newest entry that matches the given
> predicate by binary search(See `OpFindNewest.java`).
> In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> timestamp` by calculating
> the range of ledgers that may contain the target timestamp by
> `LedgerInfo#timestamp` and we don't need to scan all
> ledgers.
> However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> query entries by `BrokerEntryMetadata#index`,
> there is no more efficient way,
> we have to scan all ledgers by binary search to find the target entry.
>
> # Motivation
> Introduce per-ledger properties and we can store the extra per-ledger
> properties in the `LedgerInfo`,
> so we can query entries by `incremental index` more efficiently, say,
> `BrokerEntryMetadata#index`.
>
> PR link: https://github.com/apache/pulsar/pull/23837/files
>
> Please help review the PIP and vote.
>
> Thanks,
> Tao Jiuming(dao-jun)
>
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Tao Jiuming <da...@apache.org>.
Close the vote with 3 bindings:
- Penghui Li
- Lari Hotari
- Yunze Xu
Thanks for you review and vote!
Tao Jiuming(dao-jun)
PengHui Li <pe...@apache.org> 于2025年3月5日周三 10:57写道：
> +1 binding
>
> - Penghui
>
> On Mon, Mar 3, 2025 at 2:28 AM Tao Jiuming <da...@apache.org> wrote:
>
> > bump
> >
> > Tao Jiuming <da...@apache.org> 于2025年2月11日周二 14:37写道：
> >
> > > Hi all,
> > >
> > > I open PIP-404 to introduce per ledger properties, which will add a
> > > `properties` field to `LedgerInfo` to store extra properties for every
> > > `Ledger`.
> > >
> > > # Background knowledge
> > > As we don't have a secondary index on the Bookkeeper, so we can't query
> > > entries by message metadata efficiently.
> > > The `ManagedCursor` provides a method `asyncFindNewestMatching` to find
> > > the newest entry that matches the given
> > > predicate by binary search(See `OpFindNewest.java`).
> > > In https://github.com/apache/pulsar/pull/22792, we optimized `seeking
> by
> > > timestamp` by calculating
> > > the range of ledgers that may contain the target timestamp by
> > > `LedgerInfo#timestamp` and we don't need to scan all
> > > ledgers.
> > > However, when we enabled `AppendIndexMetadataInterceptor` and we want
> to
> > > query entries by `BrokerEntryMetadata#index`,
> > > there is no more efficient way,
> > > we have to scan all ledgers by binary search to find the target entry.
> > >
> > > # Motivation
> > > Introduce per-ledger properties and we can store the extra per-ledger
> > > properties in the `LedgerInfo`,
> > > so we can query entries by `incremental index` more efficiently, say,
> > > `BrokerEntryMetadata#index`.
> > >
> > > PR link: https://github.com/apache/pulsar/pull/23837/files
> > >
> > > Please help review the PIP and vote.
> > >
> > > Thanks,
> > > Tao Jiuming(dao-jun)
> > >
> >
>
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by PengHui Li <pe...@apache.org>.
+1 binding
- Penghui
On Mon, Mar 3, 2025 at 2:28 AM Tao Jiuming <da...@apache.org> wrote:
> bump
>
> Tao Jiuming <da...@apache.org> 于2025年2月11日周二 14:37写道：
>
> > Hi all,
> >
> > I open PIP-404 to introduce per ledger properties, which will add a
> > `properties` field to `LedgerInfo` to store extra properties for every
> > `Ledger`.
> >
> > # Background knowledge
> > As we don't have a secondary index on the Bookkeeper, so we can't query
> > entries by message metadata efficiently.
> > The `ManagedCursor` provides a method `asyncFindNewestMatching` to find
> > the newest entry that matches the given
> > predicate by binary search(See `OpFindNewest.java`).
> > In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> > timestamp` by calculating
> > the range of ledgers that may contain the target timestamp by
> > `LedgerInfo#timestamp` and we don't need to scan all
> > ledgers.
> > However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> > query entries by `BrokerEntryMetadata#index`,
> > there is no more efficient way,
> > we have to scan all ledgers by binary search to find the target entry.
> >
> > # Motivation
> > Introduce per-ledger properties and we can store the extra per-ledger
> > properties in the `LedgerInfo`,
> > so we can query entries by `incremental index` more efficiently, say,
> > `BrokerEntryMetadata#index`.
> >
> > PR link: https://github.com/apache/pulsar/pull/23837/files
> >
> > Please help review the PIP and vote.
> >
> > Thanks,
> > Tao Jiuming(dao-jun)
> >
>
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Tao Jiuming <da...@apache.org>.
bump
Tao Jiuming <da...@apache.org> 于2025年2月11日周二 14:37写道：
> Hi all,
>
> I open PIP-404 to introduce per ledger properties, which will add a
> `properties` field to `LedgerInfo` to store extra properties for every
> `Ledger`.
>
> # Background knowledge
> As we don't have a secondary index on the Bookkeeper, so we can't query
> entries by message metadata efficiently.
> The `ManagedCursor` provides a method `asyncFindNewestMatching` to find
> the newest entry that matches the given
> predicate by binary search(See `OpFindNewest.java`).
> In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> timestamp` by calculating
> the range of ledgers that may contain the target timestamp by
> `LedgerInfo#timestamp` and we don't need to scan all
> ledgers.
> However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> query entries by `BrokerEntryMetadata#index`,
> there is no more efficient way,
> we have to scan all ledgers by binary search to find the target entry.
>
> # Motivation
> Introduce per-ledger properties and we can store the extra per-ledger
> properties in the `LedgerInfo`,
> so we can query entries by `incremental index` more efficiently, say,
> `BrokerEntryMetadata#index`.
>
> PR link: https://github.com/apache/pulsar/pull/23837/files
>
> Please help review the PIP and vote.
>
> Thanks,
> Tao Jiuming(dao-jun)
>
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Tao Jiuming <da...@apache.org>.
Hi, Enrico
Thanks for your feedback.
1. For property's key-value datatype, for maximum universality, it only
supports String type. Just like `ManagedCursor#cursorProperties` and
`ManagedLedger#propertiesMap`.
2. We can add/update/remove properties during the whole ledger's life time,
all the changes will be persisted to metadata store. During use, keys of
different component types cannot be duplicated. This needs to be noted in
practical use
Enrico Olivelli <eo...@gmail.com> 于2025年2月11日周二 15:15写道：
> Tao,
> I think that the proposal is valuable.
>
> We have to clarify which datatypes are supported both for the keys and for
> the values.
> Are keys and values only raw arrays of bytes?
>
> It would be better to have a way to detect the types, so that automatic
> tools, like CLIs and UIs can represent the value in a human readable format
> or even perform some sort of filtering.
>
> Second question: are these properties writable only at ledger creation
> time? Can you update them ? Which are the rules? How does coordination
> happen ?
>
> Enrico
>
> Il Mar 11 Feb 2025, 07:38 Tao Jiuming <da...@apache.org> ha scritto:
>
> > Hi all,
> >
> > I open PIP-404 to introduce per ledger properties, which will add a
> > `properties` field to `LedgerInfo` to store extra properties for every
> > `Ledger`.
> >
> > # Background knowledge
> > As we don't have a secondary index on the Bookkeeper, so we can't query
> > entries by message metadata efficiently.
> > The `ManagedCursor` provides a method `asyncFindNewestMatching` to find
> the
> > newest entry that matches the given
> > predicate by binary search(See `OpFindNewest.java`).
> > In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> > timestamp` by calculating
> > the range of ledgers that may contain the target timestamp by
> > `LedgerInfo#timestamp` and we don't need to scan all
> > ledgers.
> > However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> > query entries by `BrokerEntryMetadata#index`,
> > there is no more efficient way,
> > we have to scan all ledgers by binary search to find the target entry.
> >
> > # Motivation
> > Introduce per-ledger properties and we can store the extra per-ledger
> > properties in the `LedgerInfo`,
> > so we can query entries by `incremental index` more efficiently, say,
> > `BrokerEntryMetadata#index`.
> >
> > PR link: https://github.com/apache/pulsar/pull/23837/files
> >
> > Please help review the PIP and vote.
> >
> > Thanks,
> > Tao Jiuming(dao-jun)
> >
>
Re: [VOTE] PIP-404： Introduce per-ledger properties
Posted by Enrico Olivelli <eo...@gmail.com>.
Tao,
I think that the proposal is valuable.
We have to clarify which datatypes are supported both for the keys and for
the values.
Are keys and values only raw arrays of bytes?
It would be better to have a way to detect the types, so that automatic
tools, like CLIs and UIs can represent the value in a human readable format
or even perform some sort of filtering.
Second question: are these properties writable only at ledger creation
time? Can you update them ? Which are the rules? How does coordination
happen ?
Enrico
Il Mar 11 Feb 2025, 07:38 Tao Jiuming <da...@apache.org> ha scritto:
> Hi all,
>
> I open PIP-404 to introduce per ledger properties, which will add a
> `properties` field to `LedgerInfo` to store extra properties for every
> `Ledger`.
>
> # Background knowledge
> As we don't have a secondary index on the Bookkeeper, so we can't query
> entries by message metadata efficiently.
> The `ManagedCursor` provides a method `asyncFindNewestMatching` to find the
> newest entry that matches the given
> predicate by binary search(See `OpFindNewest.java`).
> In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by
> timestamp` by calculating
> the range of ledgers that may contain the target timestamp by
> `LedgerInfo#timestamp` and we don't need to scan all
> ledgers.
> However, when we enabled `AppendIndexMetadataInterceptor` and we want to
> query entries by `BrokerEntryMetadata#index`,
> there is no more efficient way,
> we have to scan all ledgers by binary search to find the target entry.
>
> # Motivation
> Introduce per-ledger properties and we can store the extra per-ledger
> properties in the `LedgerInfo`,
> so we can query entries by `incremental index` more efficiently, say,
> `BrokerEntryMetadata#index`.
>
> PR link: https://github.com/apache/pulsar/pull/23837/files
>
> Please help review the PIP and vote.
>
> Thanks,
> Tao Jiuming(dao-jun)
>