You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Enrico Olivelli <eo...@gmail.com> on 2017/09/28 10:20:38 UTC
Ability to read LedgerMetadata without opening a ledger
Hi,
I am looking for an API to read ledger metadata without actually opening a
ledger.
Currently opening a ledger sets a watch on ZK and this is really expensive
and in any case the desired action is not to "open a ledger" but to access
meta information about the ledger.
My real usecase is that I want to list ledgers and filter the results using
custom metadata.
If there is no available solution I would like to file and issue and a
proposal to have a new method in the new API like
org.apache.bookkeeper.client.api.LedgerMetadata {
public long getId();
public Map<String, byte[]> getCustomMetadata();
public long getCtime();
}
org.apache.bookkeeper.client.api.BookKeeper {
LedgerMetadata getLedgerMetadata(long ledgerId);
void listLedgers(String query, Consumer<LedgerMetadata> consumer)
}
for the "query" we can define a simple "expression language"
metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
'zzzz')
we can provide a basic implementation of that language which actually works
on LedgerMetadata objects and internally we will loop over the ledgers and
apply the expression to every ledger
the basic implementation can leverage standard expression languages like EL
smarter implementations of LedgerManager will be able to narrow the search
and save resources
Thoughts ?
-- Enrico
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Ivan Kelly <iv...@apache.org>.
> BookKeeper object does not expose LedgerManager, I don't know if we want to
> expose it in the new API
LedgerManagerFactory#newLedgerManagerFactory is public now, and you
can get a ledger manager from there. Why not just use it directly for
the moment? If/when there comes a time where we break the interface,
and you need some guarantees of BC, split out an interface at _that_
point, rather than doing the work upfront for a problem that may never
happen.
-Ivan
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Enrico Olivelli <eo...@gmail.com>.
Il lun 2 ott 2017, 18:33 Sijie Guo <gu...@gmail.com> ha scritto:
> On Sun, Oct 1, 2017 at 10:11 PM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il lun 2 ott 2017, 04:06 Sijie Guo <gu...@gmail.com> ha scritto:
> >
> > > On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eo...@gmail.com>
> wrote:
> > >
> > > 2017-09-28 15:34 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> > >
> > > > - these interfaces are more an admin interface. you can get an
> > bookkeeper
> > > > admin object and then access ledger manager to fetch metadata, does
> > that
> > > > work for you?
> > > >
> > >
> > > Yes as a workaround I am taking exactly this approch...listLedgers ->
> > > getLedgerMetadata (using BK 4.5)
> > >
> > > As the project I am working on will use 4.6 and the new
> > > org.apache.bookkeeper.client.api as I need volatily durability ledgers
> > it
> > > is a pity to me that I will be stuck to use BookKeeperAdmin and
> > > LedgerMetadata
> > > only for listing ledgers
> > >
> > >
> > > We can propose the interface for bk admin. But I don't see a reason
> that
> > > listing ledgers being as part of bookkeeper.
> > >
> >
> > Ok
> >
> >
> > >
> > >
> > > > - metadata filter is just a wrapper over ledger iterator or async
> > process
> > > > ledger of ledger manager. most likely you will be the logic
> > implementer.
> > > I
> > > > don't see any strong reason we need to expose this filter as well. I
> > > would
> > > > suggesting deferring this kind of requirements when we really need
> it.
> > > >
> > >
> > > we could at least add listLedgers + getLedgerMetadata API in the new
> API
> > so
> > > that new clients won't need BookKeeperAdmin, which takes a direct ZK
> > client
> > >
> > >
> > > -1 for any direct zk access. It should be hidden behind a ledger
> manager.
> > >
> >
> > Sorry maybe I was not clear, I did not propose to make zk more visibile.
> > I wrote that I would like that new clients do not use legacy BookKeeper
> > Admin class which has explicit deps on zk
> >
> > >
> > > or a legacy o.a.b.client.BookKeper object
> > >
> > > I have a good use case, to access the list of ledgers which satisfy a
> > given
> > > condition on custom metadata, we introduced custom metadata in order to
> > > mark ledgers and to meta operations, classifications....
> > >
> > >
> > > If metadata store doesn't support predicate pushdown, you still iterate
> > all
> > > the ledgers and do the filter at your side. So I don't see a reason to
> > > support filter.
> > >
> >
> > In case of a ledgermanager which lives inside the bookie and in presence
> of
> > a big number of ledgers, say 100000, we will save a lot of network by
> > returning only a sublist of ledgers to the client.
> > In the new scenario of thin client this makes sense to me
> >
>
> It will save the network bandwidth in the thin client case. but it still
> doesn't save any things when talking to zookeeper directly.
> I am not against filtering approach. I am suggesting it doesn't buy
> applications any benefits at this moment. I would prefer proposing
> such interface after we have a better metadata store.
>
At the moment we can wait, there is no hurry.
In the future we will have a new admin API, but now let's wait for thin
client and http
Thank you all
Enrico
> >
> > >
> > > And if you are filtering custom metadata, this logic sounds tight to
> > > application logic, no?
> > >
> >
> > Yes but we can give basic support for simple filtering. For instance I am
> > usibg metadata in order to:
> > - define the application which created the ledger
> > - define a group if semantically bound ledgers
> > - add explicit references to aplication ids
> >
> > We introduced custom metada as key value pairs and I see it very simple
> to
> > create a basic but effective API to use them
> >
>
> Currently it is not more effective than using the 'ledger range iterator'
> or 'async process ledger' in the ledger manager.
> since there is already an interface for iterating ledgers and there isn't a
> metadata store supports filtering effectively, I would
> suggest doing any metadata filter api after we have a better metadata
> store.
>
> It is going be just a filter wrapper on top of 'ledger range iterator' at
> current implementation. It doesn't really buy you any benefits at this
> moment.
>
>
> >
> > >
> > >
> > > It is not blocker but if you are ok I will create an issue, but before
> > > creating an issue I would like to draft a starting idea which would
> work
> > > for the community as usual
> > >
> > >
> > > I don't see a strong value having a metadata filter, given it doesn't
> buy
> > > you any benefits at current metadata store. So I am -0 to this idea.
> > >
> >
> > As I wrote above, I will see the benefits in new implementations, like
> the
> > http API and the thin client
> >
>
> In my view, you have to design http API or thin client with filtering
> semantic first in a consistent way. Then you will know the requirement for
> what can be improved in the ledger metadata interface.
> I am seeing this is done in a reverse way.
>
>
> >
> > Enrico
> >
> > >
> > >
> > > Thanks
> > > Enrico
> > >
> > >
> > >
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eolivelli@gmail.com
> >
> > > > wrote:
> > > >
> > > > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> > > > >
> > > > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I am looking for an API to read ledger metadata without actually
> > > > opening
> > > > > a
> > > > > > ledger.
> > > > > >
> > > > > > Currently opening a ledger sets a watch on ZK and this is really
> > > > > expensive
> > > > > > and in any case the desired action is not to "open a ledger" but
> to
> > > > > access
> > > > > > meta information about the ledger.
> > > > > >
> > > > > > My real usecase is that I want to list ledgers and filter the
> > results
> > > > > using
> > > > > > custom metadata.
> > > > > >
> > > > > > If there is no available solution I would like to file and issue
> > and
> > > a
> > > > > > proposal to have a new method in the new API like
> > > > > >
> > > > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > > > public long getId();
> > > > > > public Map<String, byte[]> getCustomMetadata();
> > > > > > public long getCtime();
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I am fine with an interface for ledger metadata. It can help hide
> > the
> > > > > > implementation details.
> > > > > >
> > > > > >
> > > > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > > > >
> > > > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > > > void listLedgers(String query, Consumer<LedgerMetadata>
> > > consumer)
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I am not sure if we want this. Now, you can access the ledger
> > manager
> > > > for
> > > > > > reading and listing metadata. And this is how it was used by bk
> > > shell.
> > > > I
> > > > > > would not suggest adding this to the public API until we really
> > see a
> > > > > value
> > > > > > there.
> > > > > >
> > > > >
> > > > > BookKeeper object does not expose LedgerManager, I don't know if we
> > > want
> > > > to
> > > > > expose it in the new API
> > > > >
> > > > > we could add an API like:
> > > > >
> > > > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > > > // marker interface
> > > > > }
> > > > >
> > > > > class org.apache.bookkeeper.client.api.CustomMetadataFilter
> > implements
> > > > > LedgerMetadataFilter {
> > > > > // filter custom metadata field == value
> > > > > CustomMetadataFilter(String key, byte[] value);
> > > > > }
> > > > >
> > > > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > > > implements
> > > > > LedgerMetadataFilter {
> > > > > // apply a sequence of filters in AND
> > > > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > > > }
> > > > >
> > > > >
> > > > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> > > consumer)
> > > > >
> > > > > Does this sound better to you ? it is more simpler but clear and
> the
> > > > > implemenation will be really easy
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > for the "query" we can define a simple "expression language"
> > > > > >
> > > > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or
> > metadata.type =
> > > > > > 'zzzz')
> > > > > >
> > > > > >
> > > > > > I am not a fan of having a query language for such purpose in bk.
> > At
> > > > > least,
> > > > > > I don't see a lot of use cases that need this kind of query
> > > > capabilities.
> > > > > > If you can share more use cases about this, that would be good
> for
> > > > > > discussions.
> > > > > >
> > > > > >
> > > > > > we can provide a basic implementation of that language which
> > actually
> > > > > works
> > > > > > on LedgerMetadata objects and internally we will loop over the
> > > ledgers
> > > > > and
> > > > > > apply the expression to every ledger
> > > > > > the basic implementation can leverage standard expression
> languages
> > > > like
> > > > > EL
> > > > > >
> > > > > > smarter implementations of LedgerManager will be able to narrow
> the
> > > > > search
> > > > > > and save resources
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > -- Enrico
> > > > > >
> > > > >
> > > >
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
--
-- Enrico Olivelli
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Sijie Guo <gu...@gmail.com>.
On Sun, Oct 1, 2017 at 10:11 PM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Il lun 2 ott 2017, 04:06 Sijie Guo <gu...@gmail.com> ha scritto:
>
> > On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
> >
> > 2017-09-28 15:34 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> >
> > > - these interfaces are more an admin interface. you can get an
> bookkeeper
> > > admin object and then access ledger manager to fetch metadata, does
> that
> > > work for you?
> > >
> >
> > Yes as a workaround I am taking exactly this approch...listLedgers ->
> > getLedgerMetadata (using BK 4.5)
> >
> > As the project I am working on will use 4.6 and the new
> > org.apache.bookkeeper.client.api as I need volatily durability ledgers
> it
> > is a pity to me that I will be stuck to use BookKeeperAdmin and
> > LedgerMetadata
> > only for listing ledgers
> >
> >
> > We can propose the interface for bk admin. But I don't see a reason that
> > listing ledgers being as part of bookkeeper.
> >
>
> Ok
>
>
> >
> >
> > > - metadata filter is just a wrapper over ledger iterator or async
> process
> > > ledger of ledger manager. most likely you will be the logic
> implementer.
> > I
> > > don't see any strong reason we need to expose this filter as well. I
> > would
> > > suggesting deferring this kind of requirements when we really need it.
> > >
> >
> > we could at least add listLedgers + getLedgerMetadata API in the new API
> so
> > that new clients won't need BookKeeperAdmin, which takes a direct ZK
> client
> >
> >
> > -1 for any direct zk access. It should be hidden behind a ledger manager.
> >
>
> Sorry maybe I was not clear, I did not propose to make zk more visibile.
> I wrote that I would like that new clients do not use legacy BookKeeper
> Admin class which has explicit deps on zk
>
> >
> > or a legacy o.a.b.client.BookKeper object
> >
> > I have a good use case, to access the list of ledgers which satisfy a
> given
> > condition on custom metadata, we introduced custom metadata in order to
> > mark ledgers and to meta operations, classifications....
> >
> >
> > If metadata store doesn't support predicate pushdown, you still iterate
> all
> > the ledgers and do the filter at your side. So I don't see a reason to
> > support filter.
> >
>
> In case of a ledgermanager which lives inside the bookie and in presence of
> a big number of ledgers, say 100000, we will save a lot of network by
> returning only a sublist of ledgers to the client.
> In the new scenario of thin client this makes sense to me
>
It will save the network bandwidth in the thin client case. but it still
doesn't save any things when talking to zookeeper directly.
I am not against filtering approach. I am suggesting it doesn't buy
applications any benefits at this moment. I would prefer proposing
such interface after we have a better metadata store.
>
> >
> > And if you are filtering custom metadata, this logic sounds tight to
> > application logic, no?
> >
>
> Yes but we can give basic support for simple filtering. For instance I am
> usibg metadata in order to:
> - define the application which created the ledger
> - define a group if semantically bound ledgers
> - add explicit references to aplication ids
>
> We introduced custom metada as key value pairs and I see it very simple to
> create a basic but effective API to use them
>
Currently it is not more effective than using the 'ledger range iterator'
or 'async process ledger' in the ledger manager.
since there is already an interface for iterating ledgers and there isn't a
metadata store supports filtering effectively, I would
suggest doing any metadata filter api after we have a better metadata store.
It is going be just a filter wrapper on top of 'ledger range iterator' at
current implementation. It doesn't really buy you any benefits at this
moment.
>
> >
> >
> > It is not blocker but if you are ok I will create an issue, but before
> > creating an issue I would like to draft a starting idea which would work
> > for the community as usual
> >
> >
> > I don't see a strong value having a metadata filter, given it doesn't buy
> > you any benefits at current metadata store. So I am -0 to this idea.
> >
>
> As I wrote above, I will see the benefits in new implementations, like the
> http API and the thin client
>
In my view, you have to design http API or thin client with filtering
semantic first in a consistent way. Then you will know the requirement for
what can be improved in the ledger metadata interface.
I am seeing this is done in a reverse way.
>
> Enrico
>
> >
> >
> > Thanks
> > Enrico
> >
> >
> >
> >
> >
> > >
> > >
> > >
> > > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> > > >
> > > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi,
> > > > > I am looking for an API to read ledger metadata without actually
> > > opening
> > > > a
> > > > > ledger.
> > > > >
> > > > > Currently opening a ledger sets a watch on ZK and this is really
> > > > expensive
> > > > > and in any case the desired action is not to "open a ledger" but to
> > > > access
> > > > > meta information about the ledger.
> > > > >
> > > > > My real usecase is that I want to list ledgers and filter the
> results
> > > > using
> > > > > custom metadata.
> > > > >
> > > > > If there is no available solution I would like to file and issue
> and
> > a
> > > > > proposal to have a new method in the new API like
> > > > >
> > > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > > public long getId();
> > > > > public Map<String, byte[]> getCustomMetadata();
> > > > > public long getCtime();
> > > > > }
> > > > >
> > > > >
> > > > > I am fine with an interface for ledger metadata. It can help hide
> the
> > > > > implementation details.
> > > > >
> > > > >
> > > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > > >
> > > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > > void listLedgers(String query, Consumer<LedgerMetadata>
> > consumer)
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > > I am not sure if we want this. Now, you can access the ledger
> manager
> > > for
> > > > > reading and listing metadata. And this is how it was used by bk
> > shell.
> > > I
> > > > > would not suggest adding this to the public API until we really
> see a
> > > > value
> > > > > there.
> > > > >
> > > >
> > > > BookKeeper object does not expose LedgerManager, I don't know if we
> > want
> > > to
> > > > expose it in the new API
> > > >
> > > > we could add an API like:
> > > >
> > > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > > // marker interface
> > > > }
> > > >
> > > > class org.apache.bookkeeper.client.api.CustomMetadataFilter
> implements
> > > > LedgerMetadataFilter {
> > > > // filter custom metadata field == value
> > > > CustomMetadataFilter(String key, byte[] value);
> > > > }
> > > >
> > > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > > implements
> > > > LedgerMetadataFilter {
> > > > // apply a sequence of filters in AND
> > > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > > }
> > > >
> > > >
> > > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> > consumer)
> > > >
> > > > Does this sound better to you ? it is more simpler but clear and the
> > > > implemenation will be really easy
> > > >
> > > >
> > > >
> > > > >
> > > > >
> > > > > for the "query" we can define a simple "expression language"
> > > > >
> > > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or
> metadata.type =
> > > > > 'zzzz')
> > > > >
> > > > >
> > > > > I am not a fan of having a query language for such purpose in bk.
> At
> > > > least,
> > > > > I don't see a lot of use cases that need this kind of query
> > > capabilities.
> > > > > If you can share more use cases about this, that would be good for
> > > > > discussions.
> > > > >
> > > > >
> > > > > we can provide a basic implementation of that language which
> actually
> > > > works
> > > > > on LedgerMetadata objects and internally we will loop over the
> > ledgers
> > > > and
> > > > > apply the expression to every ledger
> > > > > the basic implementation can leverage standard expression languages
> > > like
> > > > EL
> > > > >
> > > > > smarter implementations of LedgerManager will be able to narrow the
> > > > search
> > > > > and save resources
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > -- Enrico
> > > > >
> > > >
> > >
> >
> --
>
>
> -- Enrico Olivelli
>
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Enrico Olivelli <eo...@gmail.com>.
Il lun 2 ott 2017, 04:06 Sijie Guo <gu...@gmail.com> ha scritto:
> On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
>
> 2017-09-28 15:34 GMT+02:00 Sijie Guo <gu...@gmail.com>:
>
> > - these interfaces are more an admin interface. you can get an bookkeeper
> > admin object and then access ledger manager to fetch metadata, does that
> > work for you?
> >
>
> Yes as a workaround I am taking exactly this approch...listLedgers ->
> getLedgerMetadata (using BK 4.5)
>
> As the project I am working on will use 4.6 and the new
> org.apache.bookkeeper.client.api as I need volatily durability ledgers it
> is a pity to me that I will be stuck to use BookKeeperAdmin and
> LedgerMetadata
> only for listing ledgers
>
>
> We can propose the interface for bk admin. But I don't see a reason that
> listing ledgers being as part of bookkeeper.
>
Ok
>
>
> > - metadata filter is just a wrapper over ledger iterator or async process
> > ledger of ledger manager. most likely you will be the logic implementer.
> I
> > don't see any strong reason we need to expose this filter as well. I
> would
> > suggesting deferring this kind of requirements when we really need it.
> >
>
> we could at least add listLedgers + getLedgerMetadata API in the new API so
> that new clients won't need BookKeeperAdmin, which takes a direct ZK client
>
>
> -1 for any direct zk access. It should be hidden behind a ledger manager.
>
Sorry maybe I was not clear, I did not propose to make zk more visibile.
I wrote that I would like that new clients do not use legacy BookKeeper
Admin class which has explicit deps on zk
>
> or a legacy o.a.b.client.BookKeper object
>
> I have a good use case, to access the list of ledgers which satisfy a given
> condition on custom metadata, we introduced custom metadata in order to
> mark ledgers and to meta operations, classifications....
>
>
> If metadata store doesn't support predicate pushdown, you still iterate all
> the ledgers and do the filter at your side. So I don't see a reason to
> support filter.
>
In case of a ledgermanager which lives inside the bookie and in presence of
a big number of ledgers, say 100000, we will save a lot of network by
returning only a sublist of ledgers to the client.
In the new scenario of thin client this makes sense to me
>
> And if you are filtering custom metadata, this logic sounds tight to
> application logic, no?
>
Yes but we can give basic support for simple filtering. For instance I am
usibg metadata in order to:
- define the application which created the ledger
- define a group if semantically bound ledgers
- add explicit references to aplication ids
We introduced custom metada as key value pairs and I see it very simple to
create a basic but effective API to use them
>
>
> It is not blocker but if you are ok I will create an issue, but before
> creating an issue I would like to draft a starting idea which would work
> for the community as usual
>
>
> I don't see a strong value having a metadata filter, given it doesn't buy
> you any benefits at current metadata store. So I am -0 to this idea.
>
As I wrote above, I will see the benefits in new implementations, like the
http API and the thin client
Enrico
>
>
> Thanks
> Enrico
>
>
>
>
>
> >
> >
> >
> > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> > >
> > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com>
> > wrote:
> > > >
> > > > Hi,
> > > > I am looking for an API to read ledger metadata without actually
> > opening
> > > a
> > > > ledger.
> > > >
> > > > Currently opening a ledger sets a watch on ZK and this is really
> > > expensive
> > > > and in any case the desired action is not to "open a ledger" but to
> > > access
> > > > meta information about the ledger.
> > > >
> > > > My real usecase is that I want to list ledgers and filter the results
> > > using
> > > > custom metadata.
> > > >
> > > > If there is no available solution I would like to file and issue and
> a
> > > > proposal to have a new method in the new API like
> > > >
> > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > public long getId();
> > > > public Map<String, byte[]> getCustomMetadata();
> > > > public long getCtime();
> > > > }
> > > >
> > > >
> > > > I am fine with an interface for ledger metadata. It can help hide the
> > > > implementation details.
> > > >
> > > >
> > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > >
> > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > void listLedgers(String query, Consumer<LedgerMetadata>
> consumer)
> > > >
> > > > }
> > > >
> > > >
> > > > I am not sure if we want this. Now, you can access the ledger manager
> > for
> > > > reading and listing metadata. And this is how it was used by bk
> shell.
> > I
> > > > would not suggest adding this to the public API until we really see a
> > > value
> > > > there.
> > > >
> > >
> > > BookKeeper object does not expose LedgerManager, I don't know if we
> want
> > to
> > > expose it in the new API
> > >
> > > we could add an API like:
> > >
> > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > // marker interface
> > > }
> > >
> > > class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> > > LedgerMetadataFilter {
> > > // filter custom metadata field == value
> > > CustomMetadataFilter(String key, byte[] value);
> > > }
> > >
> > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > implements
> > > LedgerMetadataFilter {
> > > // apply a sequence of filters in AND
> > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > }
> > >
> > >
> > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> consumer)
> > >
> > > Does this sound better to you ? it is more simpler but clear and the
> > > implemenation will be really easy
> > >
> > >
> > >
> > > >
> > > >
> > > > for the "query" we can define a simple "expression language"
> > > >
> > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > > > 'zzzz')
> > > >
> > > >
> > > > I am not a fan of having a query language for such purpose in bk. At
> > > least,
> > > > I don't see a lot of use cases that need this kind of query
> > capabilities.
> > > > If you can share more use cases about this, that would be good for
> > > > discussions.
> > > >
> > > >
> > > > we can provide a basic implementation of that language which actually
> > > works
> > > > on LedgerMetadata objects and internally we will loop over the
> ledgers
> > > and
> > > > apply the expression to every ledger
> > > > the basic implementation can leverage standard expression languages
> > like
> > > EL
> > > >
> > > > smarter implementations of LedgerManager will be able to narrow the
> > > search
> > > > and save resources
> > > >
> > > > Thoughts ?
> > > >
> > > > -- Enrico
> > > >
> > >
> >
>
--
-- Enrico Olivelli
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Sijie Guo <gu...@gmail.com>.
On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
2017-09-28 15:34 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> - these interfaces are more an admin interface. you can get an bookkeeper
> admin object and then access ledger manager to fetch metadata, does that
> work for you?
>
Yes as a workaround I am taking exactly this approch...listLedgers ->
getLedgerMetadata (using BK 4.5)
As the project I am working on will use 4.6 and the new
org.apache.bookkeeper.client.api as I need volatily durability ledgers it
is a pity to me that I will be stuck to use BookKeeperAdmin and
LedgerMetadata
only for listing ledgers
We can propose the interface for bk admin. But I don't see a reason that
listing ledgers being as part of bookkeeper.
> - metadata filter is just a wrapper over ledger iterator or async process
> ledger of ledger manager. most likely you will be the logic implementer. I
> don't see any strong reason we need to expose this filter as well. I would
> suggesting deferring this kind of requirements when we really need it.
>
we could at least add listLedgers + getLedgerMetadata API in the new API so
that new clients won't need BookKeeperAdmin, which takes a direct ZK client
-1 for any direct zk access. It should be hidden behind a ledger manager.
or a legacy o.a.b.client.BookKeper object
I have a good use case, to access the list of ledgers which satisfy a given
condition on custom metadata, we introduced custom metadata in order to
mark ledgers and to meta operations, classifications....
If metadata store doesn't support predicate pushdown, you still iterate all
the ledgers and do the filter at your side. So I don't see a reason to
support filter.
And if you are filtering custom metadata, this logic sounds tight to
application logic, no?
It is not blocker but if you are ok I will create an issue, but before
creating an issue I would like to draft a starting idea which would work
for the community as usual
I don't see a strong value having a metadata filter, given it doesn't buy
you any benefits at current metadata store. So I am -0 to this idea.
Thanks
Enrico
>
>
>
> On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> >
> > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com>
> wrote:
> > >
> > > Hi,
> > > I am looking for an API to read ledger metadata without actually
> opening
> > a
> > > ledger.
> > >
> > > Currently opening a ledger sets a watch on ZK and this is really
> > expensive
> > > and in any case the desired action is not to "open a ledger" but to
> > access
> > > meta information about the ledger.
> > >
> > > My real usecase is that I want to list ledgers and filter the results
> > using
> > > custom metadata.
> > >
> > > If there is no available solution I would like to file and issue and a
> > > proposal to have a new method in the new API like
> > >
> > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > public long getId();
> > > public Map<String, byte[]> getCustomMetadata();
> > > public long getCtime();
> > > }
> > >
> > >
> > > I am fine with an interface for ledger metadata. It can help hide the
> > > implementation details.
> > >
> > >
> > > org.apache.bookkeeper.client.api.BookKeeper {
> > >
> > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > void listLedgers(String query, Consumer<LedgerMetadata>
consumer)
> > >
> > > }
> > >
> > >
> > > I am not sure if we want this. Now, you can access the ledger manager
> for
> > > reading and listing metadata. And this is how it was used by bk shell.
> I
> > > would not suggest adding this to the public API until we really see a
> > value
> > > there.
> > >
> >
> > BookKeeper object does not expose LedgerManager, I don't know if we want
> to
> > expose it in the new API
> >
> > we could add an API like:
> >
> > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > // marker interface
> > }
> >
> > class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> > LedgerMetadataFilter {
> > // filter custom metadata field == value
> > CustomMetadataFilter(String key, byte[] value);
> > }
> >
> > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> implements
> > LedgerMetadataFilter {
> > // apply a sequence of filters in AND
> > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > }
> >
> >
> > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata> consumer)
> >
> > Does this sound better to you ? it is more simpler but clear and the
> > implemenation will be really easy
> >
> >
> >
> > >
> > >
> > > for the "query" we can define a simple "expression language"
> > >
> > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > > 'zzzz')
> > >
> > >
> > > I am not a fan of having a query language for such purpose in bk. At
> > least,
> > > I don't see a lot of use cases that need this kind of query
> capabilities.
> > > If you can share more use cases about this, that would be good for
> > > discussions.
> > >
> > >
> > > we can provide a basic implementation of that language which actually
> > works
> > > on LedgerMetadata objects and internally we will loop over the ledgers
> > and
> > > apply the expression to every ledger
> > > the basic implementation can leverage standard expression languages
> like
> > EL
> > >
> > > smarter implementations of LedgerManager will be able to narrow the
> > search
> > > and save resources
> > >
> > > Thoughts ?
> > >
> > > -- Enrico
> > >
> >
>
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Enrico Olivelli <eo...@gmail.com>.
2017-09-28 15:34 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> - these interfaces are more an admin interface. you can get an bookkeeper
> admin object and then access ledger manager to fetch metadata, does that
> work for you?
>
Yes as a workaround I am taking exactly this approch...listLedgers ->
getLedgerMetadata (using BK 4.5)
As the project I am working on will use 4.6 and the new
org.apache.bookkeeper.client.api as I need volatily durability ledgers it
is a pity to me that I will be stuck to use BookKeeperAdmin and
LedgerMetadata
only for listing ledgers
> - metadata filter is just a wrapper over ledger iterator or async process
> ledger of ledger manager. most likely you will be the logic implementer. I
> don't see any strong reason we need to expose this filter as well. I would
> suggesting deferring this kind of requirements when we really need it.
>
we could at least add listLedgers + getLedgerMetadata API in the new API so
that new clients won't need BookKeeperAdmin, which takes a direct ZK client
or a legacy o.a.b.client.BookKeper object
I have a good use case, to access the list of ledgers which satisfy a given
condition on custom metadata, we introduced custom metadata in order to
mark ledgers and to meta operations, classifications....
It is not blocker but if you are ok I will create an issue, but before
creating an issue I would like to draft a starting idea which would work
for the community as usual
Thanks
Enrico
>
>
>
> On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> >
> > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com>
> wrote:
> > >
> > > Hi,
> > > I am looking for an API to read ledger metadata without actually
> opening
> > a
> > > ledger.
> > >
> > > Currently opening a ledger sets a watch on ZK and this is really
> > expensive
> > > and in any case the desired action is not to "open a ledger" but to
> > access
> > > meta information about the ledger.
> > >
> > > My real usecase is that I want to list ledgers and filter the results
> > using
> > > custom metadata.
> > >
> > > If there is no available solution I would like to file and issue and a
> > > proposal to have a new method in the new API like
> > >
> > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > public long getId();
> > > public Map<String, byte[]> getCustomMetadata();
> > > public long getCtime();
> > > }
> > >
> > >
> > > I am fine with an interface for ledger metadata. It can help hide the
> > > implementation details.
> > >
> > >
> > > org.apache.bookkeeper.client.api.BookKeeper {
> > >
> > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > void listLedgers(String query, Consumer<LedgerMetadata> consumer)
> > >
> > > }
> > >
> > >
> > > I am not sure if we want this. Now, you can access the ledger manager
> for
> > > reading and listing metadata. And this is how it was used by bk shell.
> I
> > > would not suggest adding this to the public API until we really see a
> > value
> > > there.
> > >
> >
> > BookKeeper object does not expose LedgerManager, I don't know if we want
> to
> > expose it in the new API
> >
> > we could add an API like:
> >
> > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > // marker interface
> > }
> >
> > class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> > LedgerMetadataFilter {
> > // filter custom metadata field == value
> > CustomMetadataFilter(String key, byte[] value);
> > }
> >
> > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> implements
> > LedgerMetadataFilter {
> > // apply a sequence of filters in AND
> > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > }
> >
> >
> > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata> consumer)
> >
> > Does this sound better to you ? it is more simpler but clear and the
> > implemenation will be really easy
> >
> >
> >
> > >
> > >
> > > for the "query" we can define a simple "expression language"
> > >
> > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > > 'zzzz')
> > >
> > >
> > > I am not a fan of having a query language for such purpose in bk. At
> > least,
> > > I don't see a lot of use cases that need this kind of query
> capabilities.
> > > If you can share more use cases about this, that would be good for
> > > discussions.
> > >
> > >
> > > we can provide a basic implementation of that language which actually
> > works
> > > on LedgerMetadata objects and internally we will loop over the ledgers
> > and
> > > apply the expression to every ledger
> > > the basic implementation can leverage standard expression languages
> like
> > EL
> > >
> > > smarter implementations of LedgerManager will be able to narrow the
> > search
> > > and save resources
> > >
> > > Thoughts ?
> > >
> > > -- Enrico
> > >
> >
>
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Sijie Guo <gu...@gmail.com>.
- these interfaces are more an admin interface. you can get an bookkeeper
admin object and then access ledger manager to fetch metadata, does that
work for you?
- metadata filter is just a wrapper over ledger iterator or async process
ledger of ledger manager. most likely you will be the logic implementer. I
don't see any strong reason we need to expose this filter as well. I would
suggesting deferring this kind of requirements when we really need it.
On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eo...@gmail.com> wrote:
> 2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
>
> > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
> >
> > Hi,
> > I am looking for an API to read ledger metadata without actually opening
> a
> > ledger.
> >
> > Currently opening a ledger sets a watch on ZK and this is really
> expensive
> > and in any case the desired action is not to "open a ledger" but to
> access
> > meta information about the ledger.
> >
> > My real usecase is that I want to list ledgers and filter the results
> using
> > custom metadata.
> >
> > If there is no available solution I would like to file and issue and a
> > proposal to have a new method in the new API like
> >
> > org.apache.bookkeeper.client.api.LedgerMetadata {
> > public long getId();
> > public Map<String, byte[]> getCustomMetadata();
> > public long getCtime();
> > }
> >
> >
> > I am fine with an interface for ledger metadata. It can help hide the
> > implementation details.
> >
> >
> > org.apache.bookkeeper.client.api.BookKeeper {
> >
> > LedgerMetadata getLedgerMetadata(long ledgerId);
> > void listLedgers(String query, Consumer<LedgerMetadata> consumer)
> >
> > }
> >
> >
> > I am not sure if we want this. Now, you can access the ledger manager for
> > reading and listing metadata. And this is how it was used by bk shell. I
> > would not suggest adding this to the public API until we really see a
> value
> > there.
> >
>
> BookKeeper object does not expose LedgerManager, I don't know if we want to
> expose it in the new API
>
> we could add an API like:
>
> interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> // marker interface
> }
>
> class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> LedgerMetadataFilter {
> // filter custom metadata field == value
> CustomMetadataFilter(String key, byte[] value);
> }
>
> class org.apache.bookkeeper.client.api.CompositeMetadataFilter implements
> LedgerMetadataFilter {
> // apply a sequence of filters in AND
> CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> }
>
>
> void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata> consumer)
>
> Does this sound better to you ? it is more simpler but clear and the
> implemenation will be really easy
>
>
>
> >
> >
> > for the "query" we can define a simple "expression language"
> >
> > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > 'zzzz')
> >
> >
> > I am not a fan of having a query language for such purpose in bk. At
> least,
> > I don't see a lot of use cases that need this kind of query capabilities.
> > If you can share more use cases about this, that would be good for
> > discussions.
> >
> >
> > we can provide a basic implementation of that language which actually
> works
> > on LedgerMetadata objects and internally we will loop over the ledgers
> and
> > apply the expression to every ledger
> > the basic implementation can leverage standard expression languages like
> EL
> >
> > smarter implementations of LedgerManager will be able to narrow the
> search
> > and save resources
> >
> > Thoughts ?
> >
> > -- Enrico
> >
>
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Enrico Olivelli <eo...@gmail.com>.
2017-09-28 14:55 GMT+02:00 Sijie Guo <gu...@gmail.com>:
> On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
>
> Hi,
> I am looking for an API to read ledger metadata without actually opening a
> ledger.
>
> Currently opening a ledger sets a watch on ZK and this is really expensive
> and in any case the desired action is not to "open a ledger" but to access
> meta information about the ledger.
>
> My real usecase is that I want to list ledgers and filter the results using
> custom metadata.
>
> If there is no available solution I would like to file and issue and a
> proposal to have a new method in the new API like
>
> org.apache.bookkeeper.client.api.LedgerMetadata {
> public long getId();
> public Map<String, byte[]> getCustomMetadata();
> public long getCtime();
> }
>
>
> I am fine with an interface for ledger metadata. It can help hide the
> implementation details.
>
>
> org.apache.bookkeeper.client.api.BookKeeper {
>
> LedgerMetadata getLedgerMetadata(long ledgerId);
> void listLedgers(String query, Consumer<LedgerMetadata> consumer)
>
> }
>
>
> I am not sure if we want this. Now, you can access the ledger manager for
> reading and listing metadata. And this is how it was used by bk shell. I
> would not suggest adding this to the public API until we really see a value
> there.
>
BookKeeper object does not expose LedgerManager, I don't know if we want to
expose it in the new API
we could add an API like:
interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
// marker interface
}
class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
LedgerMetadataFilter {
// filter custom metadata field == value
CustomMetadataFilter(String key, byte[] value);
}
class org.apache.bookkeeper.client.api.CompositeMetadataFilter implements
LedgerMetadataFilter {
// apply a sequence of filters in AND
CompositeMetadataFilter(LedgerMetadataFilter ... filters);
}
void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata> consumer)
Does this sound better to you ? it is more simpler but clear and the
implemenation will be really easy
>
>
> for the "query" we can define a simple "expression language"
>
> metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> 'zzzz')
>
>
> I am not a fan of having a query language for such purpose in bk. At least,
> I don't see a lot of use cases that need this kind of query capabilities.
> If you can share more use cases about this, that would be good for
> discussions.
>
>
> we can provide a basic implementation of that language which actually works
> on LedgerMetadata objects and internally we will loop over the ledgers and
> apply the expression to every ledger
> the basic implementation can leverage standard expression languages like EL
>
> smarter implementations of LedgerManager will be able to narrow the search
> and save resources
>
> Thoughts ?
>
> -- Enrico
>
Re: Ability to read LedgerMetadata without opening a ledger
Posted by Sijie Guo <gu...@gmail.com>.
On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eo...@gmail.com> wrote:
Hi,
I am looking for an API to read ledger metadata without actually opening a
ledger.
Currently opening a ledger sets a watch on ZK and this is really expensive
and in any case the desired action is not to "open a ledger" but to access
meta information about the ledger.
My real usecase is that I want to list ledgers and filter the results using
custom metadata.
If there is no available solution I would like to file and issue and a
proposal to have a new method in the new API like
org.apache.bookkeeper.client.api.LedgerMetadata {
public long getId();
public Map<String, byte[]> getCustomMetadata();
public long getCtime();
}
I am fine with an interface for ledger metadata. It can help hide the
implementation details.
org.apache.bookkeeper.client.api.BookKeeper {
LedgerMetadata getLedgerMetadata(long ledgerId);
void listLedgers(String query, Consumer<LedgerMetadata> consumer)
}
I am not sure if we want this. Now, you can access the ledger manager for
reading and listing metadata. And this is how it was used by bk shell. I
would not suggest adding this to the public API until we really see a value
there.
for the "query" we can define a simple "expression language"
metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
'zzzz')
I am not a fan of having a query language for such purpose in bk. At least,
I don't see a lot of use cases that need this kind of query capabilities.
If you can share more use cases about this, that would be good for
discussions.
we can provide a basic implementation of that language which actually works
on LedgerMetadata objects and internally we will loop over the ledgers and
apply the expression to every ledger
the basic implementation can leverage standard expression languages like EL
smarter implementations of LedgerManager will be able to narrow the search
and save resources
Thoughts ?
-- Enrico