You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Jia Zhai <zh...@gmail.com> on 2017/09/12 15:00:43 UTC
[DISCUSS] BP-17: define BookKeeper public http endpoints
Hi all,
Based on Github #278 <https://github.com/apache/bookkeeper/pull/278>, I
have just posted a proposal regarding define BookKeeper public http
endpoints:
https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-17%3A+Define+BookKeeper+public+http+endpoints
Github #278 <https://github.com/apache/bookkeeper/pull/278> introduces
BookKeeper Http Endpoint module. However there are only two endpoints,
which is “/heartbeat” and “/api/config/serverconfig”, defined in #278. In
order to fully leverage the http modules, The goal of this BP is to add
more endpoints to this modules.
Any comments are welcome and appreciated.
Thanks.
-Jia
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Ivan Kelly <iv...@apache.org>.
+1
On Tue, Sep 19, 2017 at 6:44 PM, Yiming Zang <yz...@twitter.com.invalid> wrote:
> LGTM +1
>
> On Tue, Sep 19, 2017 at 8:26 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
>> Looks good to me.
>> Enrico
>>
>> On mar 19 set 2017, 17:22 Jia Zhai <zh...@gmail.com> wrote:
>>
>> > Any thoughts or comments on this. :)
>> > If not, would like to mark this BP approved. And we will prepare an
>> > initial PR for detailed discussion and comments.
>> >
>> > On Wed, Sep 13, 2017 at 4:49 PM, Sijie Guo <gu...@gmail.com> wrote:
>> >
>> >> On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eo...@gmail.com>
>> >> wrote:
>> >>
>> >> >
>> >> >
>> >> > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
>> >> >
>> >> >> Thanks a lot for your time Yiming and Enrico. :)
>> >> >>
>> >> >> Regarding the security, we could do it in a separate BP, and make
>> >> this BP
>> >> >> more focus on filling up the useful endpoints. How about it?
>> >> >>
>> >> >
>> >> > OK, but please consider that the more utility you add to the endpoint
>> >> the
>> >> > more users will want to enable it and so security will be a blocker
>> >> issue
>> >> >
>> >>
>> >> My take on this - we all know security is important for any
>> communication
>> >> to bookies.
>> >>
>> >> However, security is a big different scope of problems to address. we
>> >> shouldn't put everything in one big BP. I'd suggest us focusing on the
>> >> problems
>> >> that a BP tends to address, defer other things to separate BPs.
>> Otherwise
>> >> we can't land any changes quickly.
>> >>
>> >>
>> >> >
>> >> >
>> >> >>
>> >> >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <
>> eolivelli@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >> > Hi Jia,
>> >> >> > I am OK with the idea of having standard HTTP API, this will help
>> >> >> > development of (non-Java) tools.
>> >> >> >
>> >> >> > It is not clear to me if we are going to add an http API useful for
>> >> >> > "managing bookies" or a new HTTP REST-like Client API to
>> BookKeeper.
>> >> >> > I am referring to the fact that in the proposal there are API calls
>> >> to
>> >> >> > create ledgers and to read data.
>> >> >> > I think we should separate this two aspects and maybe it is better
>> to
>> >> >> > address the 'bookie management' first, which is the work that
>> Yiming
>> >> >> > started.
>> >> >> >
>> >> >> [jia] It is targeting for the admin portal. the existence of the
>> ledger
>> >> >> api
>> >> >> is just to simplify debugging or operations. we can eliminate the
>> >> ‘create’
>> >> >> endpoint.
>> >> >>
>> >> >
>> >> > OK so are you already thinking about specific tools to manage bookies
>> >> > without the bookie shell
>> >> >
>> >>
>> >> We would like to integrate this for schedulers (e.g. k8s). so we can
>> >> deploy
>> >> and operate bookkeeper easily
>> >> in those environments.
>> >>
>> >>
>> >> >
>> >> >
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Another point very important to me is that if we are going to
>> >> introduce
>> >> >> > management operations via http we have to take into account
>> >> security, at
>> >> >> > least TLS and some kind of authentication.
>> >> >> > About TLS it is very simple to achieve this, every HTTP server
>> >> supports
>> >> >> TLS
>> >> >> > (https)
>> >> >> > About authentication the problem is not so simple, Kerberos on
>> HTTPs
>> >> is
>> >> >> > very complex, but we need to introduce some auth mechanism.
>> >> >> > IMHO We can require the 'http server implementation' to implement
>> >> >> security
>> >> >> > but out of the box we have to supply basic support at least for one
>> >> >> > provider bundled in the distribution package.
>> >> >> >
>> >> >> [jia] The BP focuses on filling up the useful endpoints. The security
>> >> will
>> >> >> be a separate BP.
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Some API can return very large result sets like 'list ledgers',
>> >> actually
>> >> >> > the HTTP Server subsystem exchanges strings in memory, we will need
>> >> to
>> >> >> > introduce some more smart way because it would be easy to bring
>> down
>> >> the
>> >> >> > bookie just by calling that API multiple times concurrencly (it is
>> >> just
>> >> >> an
>> >> >> > example)
>> >> >> >
>> >> >> [jia] We could add pagination into all the `list` api.
>> >> >>
>> >> >
>> >> > Yes but it will be really tricky to implement, but please do it
>> >> >
>> >> >
>> >> >>
>> >> >> >
>> >> >> > IMHO The 'create ledger' API is not useful, due to the design of
>> BK
>> >> you
>> >> >> > have to create a ledger and then write immediately to it, I think
>> >> that
>> >> >> such
>> >> >> > an API should allow the client to stream the contents of the ledger
>> >> in
>> >> >> the
>> >> >> > HTTP body at least. But I think that a more stateful http API needs
>> >> to
>> >> >> be
>> >> >> > designed to implement a pure Http client
>> >> >> >
>> >> >> [jia] Thanks, we are not planning to implement an http client. we
>> will
>> >> >> remove ‘create’ here.
>> >> >>
>> >> >
>> >> > OK, thanks
>> >> >
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Maybe we should add a more narrowed motivation and add the only
>> >> useful
>> >> >> APIs
>> >> >> > to address those issues.
>> >> >> > For instance in my company we would like to start creating a
>> >> BookKeeper
>> >> >> Web
>> >> >> > UI, so we need some way to talk to bookies and exchange data, but
>> in
>> >> >> this
>> >> >> > case I am interested in asking to bookies their status and the
>> >> effective
>> >> >> > contents of the bookie
>> >> >> >
>> >> >> [jia] The BP proposes adding a standard naming convention for adding
>> >> admin
>> >> >> endpoints. Feel free to propose the endpoints you would like to
>> appear
>> >> in
>> >> >> the http admin portal.
>> >> >>
>> >> >
>> >> > The ones you wrote are enough complete for me, I would like to have
>> >> > read-only operation in order to have a global view on the status of
>> the
>> >> > system.
>> >> >
>> >> > -- Enrico
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > -- Enrico
>> >> >> >
>> >> >> >
>> >> >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
>> >> >> >
>> >> >> > > Sure, I think the current HTTP endpoints in Twitter are only
>> >> designed
>> >> >> for
>> >> >> > > Twitter specific, such as check quorum loss, check rack/region
>> >> >> diversity.
>> >> >> > > So the endpoints convention in Twitter are not the same as in the
>> >> >> > proposal.
>> >> >> > > I think it would be great to have an agreement on the API naming
>> >> >> design,
>> >> >> > so
>> >> >> > > I like the API design in the proposal, I think the proposal looks
>> >> >> good to
>> >> >> > > me.
>> >> >> > >
>> >> >> > > Besides, we're currently only using GET functionalities in
>> Twitter,
>> >> >> but I
>> >> >> > > notice there're a lot of POST and PUT APIs in the proposal which
>> >> could
>> >> >> > > change the bookie state or trigger some heavy workload. These
>> APIs
>> >> >> looks
>> >> >> > a
>> >> >> > > bit risky to me if we don't have any authentication enabled (in
>> >> >> Twitter).
>> >> >> > >
>> >> >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com>
>> >> >> wrote:
>> >> >> > >
>> >> >> > > > + Yiming
>> >> >> > > >
>> >> >> > > > Yiming, if you have time, please take a look at this BP. let's
>> >> see
>> >> >> if
>> >> >> > > > there are any conflicts with those you added for autorecovery.
>> >> >> > > >
>> >> >> > > > - Sijie
>> >> >> > > >
>> >> >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zhaijia03@gmail.com
>> >
>> >> >> wrote:
>> >> >> > > >
>> >> >> > > >> Hi all,
>> >> >> > > >>
>> >> >> > > >> Based on Github #278 <https://github.com/apache/boo
>> >> >> kkeeper/pull/278>,
>> >> >> > > I
>> >> >> > > >> have just posted a proposal regarding define BookKeeper public
>> >> http
>> >> >> > > >> endpoints:
>> >> >> > > >>
>> >> >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
>> >> >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
>> >> >> > introduces
>> >> >> > > >> BookKeeper Http Endpoint module. However there are only two
>> >> >> endpoints,
>> >> >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined
>> in
>> >> >> #278.
>> >> >> > > In
>> >> >> > > >> order to fully leverage the http modules, The goal of this BP
>> >> is to
>> >> >> > add
>> >> >> > > >> more endpoints to this modules.
>> >> >> > > >>
>> >> >> > > >> Any comments are welcome and appreciated.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> Thanks.
>> >> >> > > >>
>> >> >> > > >> -Jia
>> >> >> > > >>
>> >> >> > > >
>> >> >> > > >
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >>
>> >
>> > --
>>
>>
>> -- Enrico Olivelli
>>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Yiming Zang <yz...@twitter.com.INVALID>.
LGTM +1
On Tue, Sep 19, 2017 at 8:26 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Looks good to me.
> Enrico
>
> On mar 19 set 2017, 17:22 Jia Zhai <zh...@gmail.com> wrote:
>
> > Any thoughts or comments on this. :)
> > If not, would like to mark this BP approved. And we will prepare an
> > initial PR for detailed discussion and comments.
> >
> > On Wed, Sep 13, 2017 at 4:49 PM, Sijie Guo <gu...@gmail.com> wrote:
> >
> >> On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eo...@gmail.com>
> >> wrote:
> >>
> >> >
> >> >
> >> > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
> >> >
> >> >> Thanks a lot for your time Yiming and Enrico. :)
> >> >>
> >> >> Regarding the security, we could do it in a separate BP, and make
> >> this BP
> >> >> more focus on filling up the useful endpoints. How about it?
> >> >>
> >> >
> >> > OK, but please consider that the more utility you add to the endpoint
> >> the
> >> > more users will want to enable it and so security will be a blocker
> >> issue
> >> >
> >>
> >> My take on this - we all know security is important for any
> communication
> >> to bookies.
> >>
> >> However, security is a big different scope of problems to address. we
> >> shouldn't put everything in one big BP. I'd suggest us focusing on the
> >> problems
> >> that a BP tends to address, defer other things to separate BPs.
> Otherwise
> >> we can't land any changes quickly.
> >>
> >>
> >> >
> >> >
> >> >>
> >> >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <
> eolivelli@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Jia,
> >> >> > I am OK with the idea of having standard HTTP API, this will help
> >> >> > development of (non-Java) tools.
> >> >> >
> >> >> > It is not clear to me if we are going to add an http API useful for
> >> >> > "managing bookies" or a new HTTP REST-like Client API to
> BookKeeper.
> >> >> > I am referring to the fact that in the proposal there are API calls
> >> to
> >> >> > create ledgers and to read data.
> >> >> > I think we should separate this two aspects and maybe it is better
> to
> >> >> > address the 'bookie management' first, which is the work that
> Yiming
> >> >> > started.
> >> >> >
> >> >> [jia] It is targeting for the admin portal. the existence of the
> ledger
> >> >> api
> >> >> is just to simplify debugging or operations. we can eliminate the
> >> ‘create’
> >> >> endpoint.
> >> >>
> >> >
> >> > OK so are you already thinking about specific tools to manage bookies
> >> > without the bookie shell
> >> >
> >>
> >> We would like to integrate this for schedulers (e.g. k8s). so we can
> >> deploy
> >> and operate bookkeeper easily
> >> in those environments.
> >>
> >>
> >> >
> >> >
> >> >>
> >> >>
> >> >> >
> >> >> > Another point very important to me is that if we are going to
> >> introduce
> >> >> > management operations via http we have to take into account
> >> security, at
> >> >> > least TLS and some kind of authentication.
> >> >> > About TLS it is very simple to achieve this, every HTTP server
> >> supports
> >> >> TLS
> >> >> > (https)
> >> >> > About authentication the problem is not so simple, Kerberos on
> HTTPs
> >> is
> >> >> > very complex, but we need to introduce some auth mechanism.
> >> >> > IMHO We can require the 'http server implementation' to implement
> >> >> security
> >> >> > but out of the box we have to supply basic support at least for one
> >> >> > provider bundled in the distribution package.
> >> >> >
> >> >> [jia] The BP focuses on filling up the useful endpoints. The security
> >> will
> >> >> be a separate BP.
> >> >>
> >> >>
> >> >> >
> >> >> > Some API can return very large result sets like 'list ledgers',
> >> actually
> >> >> > the HTTP Server subsystem exchanges strings in memory, we will need
> >> to
> >> >> > introduce some more smart way because it would be easy to bring
> down
> >> the
> >> >> > bookie just by calling that API multiple times concurrencly (it is
> >> just
> >> >> an
> >> >> > example)
> >> >> >
> >> >> [jia] We could add pagination into all the `list` api.
> >> >>
> >> >
> >> > Yes but it will be really tricky to implement, but please do it
> >> >
> >> >
> >> >>
> >> >> >
> >> >> > IMHO The 'create ledger' API is not useful, due to the design of
> BK
> >> you
> >> >> > have to create a ledger and then write immediately to it, I think
> >> that
> >> >> such
> >> >> > an API should allow the client to stream the contents of the ledger
> >> in
> >> >> the
> >> >> > HTTP body at least. But I think that a more stateful http API needs
> >> to
> >> >> be
> >> >> > designed to implement a pure Http client
> >> >> >
> >> >> [jia] Thanks, we are not planning to implement an http client. we
> will
> >> >> remove ‘create’ here.
> >> >>
> >> >
> >> > OK, thanks
> >> >
> >> >>
> >> >>
> >> >> >
> >> >> > Maybe we should add a more narrowed motivation and add the only
> >> useful
> >> >> APIs
> >> >> > to address those issues.
> >> >> > For instance in my company we would like to start creating a
> >> BookKeeper
> >> >> Web
> >> >> > UI, so we need some way to talk to bookies and exchange data, but
> in
> >> >> this
> >> >> > case I am interested in asking to bookies their status and the
> >> effective
> >> >> > contents of the bookie
> >> >> >
> >> >> [jia] The BP proposes adding a standard naming convention for adding
> >> admin
> >> >> endpoints. Feel free to propose the endpoints you would like to
> appear
> >> in
> >> >> the http admin portal.
> >> >>
> >> >
> >> > The ones you wrote are enough complete for me, I would like to have
> >> > read-only operation in order to have a global view on the status of
> the
> >> > system.
> >> >
> >> > -- Enrico
> >> >
> >> >
> >> >
> >> >>
> >> >>
> >> >> >
> >> >> > -- Enrico
> >> >> >
> >> >> >
> >> >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
> >> >> >
> >> >> > > Sure, I think the current HTTP endpoints in Twitter are only
> >> designed
> >> >> for
> >> >> > > Twitter specific, such as check quorum loss, check rack/region
> >> >> diversity.
> >> >> > > So the endpoints convention in Twitter are not the same as in the
> >> >> > proposal.
> >> >> > > I think it would be great to have an agreement on the API naming
> >> >> design,
> >> >> > so
> >> >> > > I like the API design in the proposal, I think the proposal looks
> >> >> good to
> >> >> > > me.
> >> >> > >
> >> >> > > Besides, we're currently only using GET functionalities in
> Twitter,
> >> >> but I
> >> >> > > notice there're a lot of POST and PUT APIs in the proposal which
> >> could
> >> >> > > change the bookie state or trigger some heavy workload. These
> APIs
> >> >> looks
> >> >> > a
> >> >> > > bit risky to me if we don't have any authentication enabled (in
> >> >> Twitter).
> >> >> > >
> >> >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com>
> >> >> wrote:
> >> >> > >
> >> >> > > > + Yiming
> >> >> > > >
> >> >> > > > Yiming, if you have time, please take a look at this BP. let's
> >> see
> >> >> if
> >> >> > > > there are any conflicts with those you added for autorecovery.
> >> >> > > >
> >> >> > > > - Sijie
> >> >> > > >
> >> >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zhaijia03@gmail.com
> >
> >> >> wrote:
> >> >> > > >
> >> >> > > >> Hi all,
> >> >> > > >>
> >> >> > > >> Based on Github #278 <https://github.com/apache/boo
> >> >> kkeeper/pull/278>,
> >> >> > > I
> >> >> > > >> have just posted a proposal regarding define BookKeeper public
> >> http
> >> >> > > >> endpoints:
> >> >> > > >>
> >> >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
> >> >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
> >> >> > introduces
> >> >> > > >> BookKeeper Http Endpoint module. However there are only two
> >> >> endpoints,
> >> >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined
> in
> >> >> #278.
> >> >> > > In
> >> >> > > >> order to fully leverage the http modules, The goal of this BP
> >> is to
> >> >> > add
> >> >> > > >> more endpoints to this modules.
> >> >> > > >>
> >> >> > > >> Any comments are welcome and appreciated.
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> Thanks.
> >> >> > > >>
> >> >> > > >> -Jia
> >> >> > > >>
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> > --
>
>
> -- Enrico Olivelli
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Enrico Olivelli <eo...@gmail.com>.
Looks good to me.
Enrico
On mar 19 set 2017, 17:22 Jia Zhai <zh...@gmail.com> wrote:
> Any thoughts or comments on this. :)
> If not, would like to mark this BP approved. And we will prepare an
> initial PR for detailed discussion and comments.
>
> On Wed, Sep 13, 2017 at 4:49 PM, Sijie Guo <gu...@gmail.com> wrote:
>
>> On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> >
>> >
>> > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
>> >
>> >> Thanks a lot for your time Yiming and Enrico. :)
>> >>
>> >> Regarding the security, we could do it in a separate BP, and make
>> this BP
>> >> more focus on filling up the useful endpoints. How about it?
>> >>
>> >
>> > OK, but please consider that the more utility you add to the endpoint
>> the
>> > more users will want to enable it and so security will be a blocker
>> issue
>> >
>>
>> My take on this - we all know security is important for any communication
>> to bookies.
>>
>> However, security is a big different scope of problems to address. we
>> shouldn't put everything in one big BP. I'd suggest us focusing on the
>> problems
>> that a BP tends to address, defer other things to separate BPs. Otherwise
>> we can't land any changes quickly.
>>
>>
>> >
>> >
>> >>
>> >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eo...@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Jia,
>> >> > I am OK with the idea of having standard HTTP API, this will help
>> >> > development of (non-Java) tools.
>> >> >
>> >> > It is not clear to me if we are going to add an http API useful for
>> >> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper.
>> >> > I am referring to the fact that in the proposal there are API calls
>> to
>> >> > create ledgers and to read data.
>> >> > I think we should separate this two aspects and maybe it is better to
>> >> > address the 'bookie management' first, which is the work that Yiming
>> >> > started.
>> >> >
>> >> [jia] It is targeting for the admin portal. the existence of the ledger
>> >> api
>> >> is just to simplify debugging or operations. we can eliminate the
>> ‘create’
>> >> endpoint.
>> >>
>> >
>> > OK so are you already thinking about specific tools to manage bookies
>> > without the bookie shell
>> >
>>
>> We would like to integrate this for schedulers (e.g. k8s). so we can
>> deploy
>> and operate bookkeeper easily
>> in those environments.
>>
>>
>> >
>> >
>> >>
>> >>
>> >> >
>> >> > Another point very important to me is that if we are going to
>> introduce
>> >> > management operations via http we have to take into account
>> security, at
>> >> > least TLS and some kind of authentication.
>> >> > About TLS it is very simple to achieve this, every HTTP server
>> supports
>> >> TLS
>> >> > (https)
>> >> > About authentication the problem is not so simple, Kerberos on HTTPs
>> is
>> >> > very complex, but we need to introduce some auth mechanism.
>> >> > IMHO We can require the 'http server implementation' to implement
>> >> security
>> >> > but out of the box we have to supply basic support at least for one
>> >> > provider bundled in the distribution package.
>> >> >
>> >> [jia] The BP focuses on filling up the useful endpoints. The security
>> will
>> >> be a separate BP.
>> >>
>> >>
>> >> >
>> >> > Some API can return very large result sets like 'list ledgers',
>> actually
>> >> > the HTTP Server subsystem exchanges strings in memory, we will need
>> to
>> >> > introduce some more smart way because it would be easy to bring down
>> the
>> >> > bookie just by calling that API multiple times concurrencly (it is
>> just
>> >> an
>> >> > example)
>> >> >
>> >> [jia] We could add pagination into all the `list` api.
>> >>
>> >
>> > Yes but it will be really tricky to implement, but please do it
>> >
>> >
>> >>
>> >> >
>> >> > IMHO The 'create ledger' API is not useful, due to the design of BK
>> you
>> >> > have to create a ledger and then write immediately to it, I think
>> that
>> >> such
>> >> > an API should allow the client to stream the contents of the ledger
>> in
>> >> the
>> >> > HTTP body at least. But I think that a more stateful http API needs
>> to
>> >> be
>> >> > designed to implement a pure Http client
>> >> >
>> >> [jia] Thanks, we are not planning to implement an http client. we will
>> >> remove ‘create’ here.
>> >>
>> >
>> > OK, thanks
>> >
>> >>
>> >>
>> >> >
>> >> > Maybe we should add a more narrowed motivation and add the only
>> useful
>> >> APIs
>> >> > to address those issues.
>> >> > For instance in my company we would like to start creating a
>> BookKeeper
>> >> Web
>> >> > UI, so we need some way to talk to bookies and exchange data, but in
>> >> this
>> >> > case I am interested in asking to bookies their status and the
>> effective
>> >> > contents of the bookie
>> >> >
>> >> [jia] The BP proposes adding a standard naming convention for adding
>> admin
>> >> endpoints. Feel free to propose the endpoints you would like to appear
>> in
>> >> the http admin portal.
>> >>
>> >
>> > The ones you wrote are enough complete for me, I would like to have
>> > read-only operation in order to have a global view on the status of the
>> > system.
>> >
>> > -- Enrico
>> >
>> >
>> >
>> >>
>> >>
>> >> >
>> >> > -- Enrico
>> >> >
>> >> >
>> >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
>> >> >
>> >> > > Sure, I think the current HTTP endpoints in Twitter are only
>> designed
>> >> for
>> >> > > Twitter specific, such as check quorum loss, check rack/region
>> >> diversity.
>> >> > > So the endpoints convention in Twitter are not the same as in the
>> >> > proposal.
>> >> > > I think it would be great to have an agreement on the API naming
>> >> design,
>> >> > so
>> >> > > I like the API design in the proposal, I think the proposal looks
>> >> good to
>> >> > > me.
>> >> > >
>> >> > > Besides, we're currently only using GET functionalities in Twitter,
>> >> but I
>> >> > > notice there're a lot of POST and PUT APIs in the proposal which
>> could
>> >> > > change the bookie state or trigger some heavy workload. These APIs
>> >> looks
>> >> > a
>> >> > > bit risky to me if we don't have any authentication enabled (in
>> >> Twitter).
>> >> > >
>> >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com>
>> >> wrote:
>> >> > >
>> >> > > > + Yiming
>> >> > > >
>> >> > > > Yiming, if you have time, please take a look at this BP. let's
>> see
>> >> if
>> >> > > > there are any conflicts with those you added for autorecovery.
>> >> > > >
>> >> > > > - Sijie
>> >> > > >
>> >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com>
>> >> wrote:
>> >> > > >
>> >> > > >> Hi all,
>> >> > > >>
>> >> > > >> Based on Github #278 <https://github.com/apache/boo
>> >> kkeeper/pull/278>,
>> >> > > I
>> >> > > >> have just posted a proposal regarding define BookKeeper public
>> http
>> >> > > >> endpoints:
>> >> > > >>
>> >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
>> >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
>> >> > > >>
>> >> > > >>
>> >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
>> >> > introduces
>> >> > > >> BookKeeper Http Endpoint module. However there are only two
>> >> endpoints,
>> >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in
>> >> #278.
>> >> > > In
>> >> > > >> order to fully leverage the http modules, The goal of this BP
>> is to
>> >> > add
>> >> > > >> more endpoints to this modules.
>> >> > > >>
>> >> > > >> Any comments are welcome and appreciated.
>> >> > > >>
>> >> > > >>
>> >> > > >> Thanks.
>> >> > > >>
>> >> > > >> -Jia
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
> --
-- Enrico Olivelli
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Jia Zhai <zh...@gmail.com>.
Any thoughts or comments on this. :)
If not, would like to mark this BP approved. And we will prepare an initial
PR for detailed discussion and comments.
On Wed, Sep 13, 2017 at 4:49 PM, Sijie Guo <gu...@gmail.com> wrote:
> On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> >
> >
> > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
> >
> >> Thanks a lot for your time Yiming and Enrico. :)
> >>
> >> Regarding the security, we could do it in a separate BP, and make this
> BP
> >> more focus on filling up the useful endpoints. How about it?
> >>
> >
> > OK, but please consider that the more utility you add to the endpoint the
> > more users will want to enable it and so security will be a blocker issue
> >
>
> My take on this - we all know security is important for any communication
> to bookies.
>
> However, security is a big different scope of problems to address. we
> shouldn't put everything in one big BP. I'd suggest us focusing on the
> problems
> that a BP tends to address, defer other things to separate BPs. Otherwise
> we can't land any changes quickly.
>
>
> >
> >
> >>
> >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eo...@gmail.com>
> >> wrote:
> >>
> >> > Hi Jia,
> >> > I am OK with the idea of having standard HTTP API, this will help
> >> > development of (non-Java) tools.
> >> >
> >> > It is not clear to me if we are going to add an http API useful for
> >> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper.
> >> > I am referring to the fact that in the proposal there are API calls to
> >> > create ledgers and to read data.
> >> > I think we should separate this two aspects and maybe it is better to
> >> > address the 'bookie management' first, which is the work that Yiming
> >> > started.
> >> >
> >> [jia] It is targeting for the admin portal. the existence of the ledger
> >> api
> >> is just to simplify debugging or operations. we can eliminate the
> ‘create’
> >> endpoint.
> >>
> >
> > OK so are you already thinking about specific tools to manage bookies
> > without the bookie shell
> >
>
> We would like to integrate this for schedulers (e.g. k8s). so we can deploy
> and operate bookkeeper easily
> in those environments.
>
>
> >
> >
> >>
> >>
> >> >
> >> > Another point very important to me is that if we are going to
> introduce
> >> > management operations via http we have to take into account security,
> at
> >> > least TLS and some kind of authentication.
> >> > About TLS it is very simple to achieve this, every HTTP server
> supports
> >> TLS
> >> > (https)
> >> > About authentication the problem is not so simple, Kerberos on HTTPs
> is
> >> > very complex, but we need to introduce some auth mechanism.
> >> > IMHO We can require the 'http server implementation' to implement
> >> security
> >> > but out of the box we have to supply basic support at least for one
> >> > provider bundled in the distribution package.
> >> >
> >> [jia] The BP focuses on filling up the useful endpoints. The security
> will
> >> be a separate BP.
> >>
> >>
> >> >
> >> > Some API can return very large result sets like 'list ledgers',
> actually
> >> > the HTTP Server subsystem exchanges strings in memory, we will need to
> >> > introduce some more smart way because it would be easy to bring down
> the
> >> > bookie just by calling that API multiple times concurrencly (it is
> just
> >> an
> >> > example)
> >> >
> >> [jia] We could add pagination into all the `list` api.
> >>
> >
> > Yes but it will be really tricky to implement, but please do it
> >
> >
> >>
> >> >
> >> > IMHO The 'create ledger' API is not useful, due to the design of BK
> you
> >> > have to create a ledger and then write immediately to it, I think that
> >> such
> >> > an API should allow the client to stream the contents of the ledger in
> >> the
> >> > HTTP body at least. But I think that a more stateful http API needs to
> >> be
> >> > designed to implement a pure Http client
> >> >
> >> [jia] Thanks, we are not planning to implement an http client. we will
> >> remove ‘create’ here.
> >>
> >
> > OK, thanks
> >
> >>
> >>
> >> >
> >> > Maybe we should add a more narrowed motivation and add the only useful
> >> APIs
> >> > to address those issues.
> >> > For instance in my company we would like to start creating a
> BookKeeper
> >> Web
> >> > UI, so we need some way to talk to bookies and exchange data, but in
> >> this
> >> > case I am interested in asking to bookies their status and the
> effective
> >> > contents of the bookie
> >> >
> >> [jia] The BP proposes adding a standard naming convention for adding
> admin
> >> endpoints. Feel free to propose the endpoints you would like to appear
> in
> >> the http admin portal.
> >>
> >
> > The ones you wrote are enough complete for me, I would like to have
> > read-only operation in order to have a global view on the status of the
> > system.
> >
> > -- Enrico
> >
> >
> >
> >>
> >>
> >> >
> >> > -- Enrico
> >> >
> >> >
> >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
> >> >
> >> > > Sure, I think the current HTTP endpoints in Twitter are only
> designed
> >> for
> >> > > Twitter specific, such as check quorum loss, check rack/region
> >> diversity.
> >> > > So the endpoints convention in Twitter are not the same as in the
> >> > proposal.
> >> > > I think it would be great to have an agreement on the API naming
> >> design,
> >> > so
> >> > > I like the API design in the proposal, I think the proposal looks
> >> good to
> >> > > me.
> >> > >
> >> > > Besides, we're currently only using GET functionalities in Twitter,
> >> but I
> >> > > notice there're a lot of POST and PUT APIs in the proposal which
> could
> >> > > change the bookie state or trigger some heavy workload. These APIs
> >> looks
> >> > a
> >> > > bit risky to me if we don't have any authentication enabled (in
> >> Twitter).
> >> > >
> >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com>
> >> wrote:
> >> > >
> >> > > > + Yiming
> >> > > >
> >> > > > Yiming, if you have time, please take a look at this BP. let's see
> >> if
> >> > > > there are any conflicts with those you added for autorecovery.
> >> > > >
> >> > > > - Sijie
> >> > > >
> >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com>
> >> wrote:
> >> > > >
> >> > > >> Hi all,
> >> > > >>
> >> > > >> Based on Github #278 <https://github.com/apache/boo
> >> kkeeper/pull/278>,
> >> > > I
> >> > > >> have just posted a proposal regarding define BookKeeper public
> http
> >> > > >> endpoints:
> >> > > >>
> >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
> >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
> >> > > >>
> >> > > >>
> >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
> >> > introduces
> >> > > >> BookKeeper Http Endpoint module. However there are only two
> >> endpoints,
> >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in
> >> #278.
> >> > > In
> >> > > >> order to fully leverage the http modules, The goal of this BP is
> to
> >> > add
> >> > > >> more endpoints to this modules.
> >> > > >>
> >> > > >> Any comments are welcome and appreciated.
> >> > > >>
> >> > > >>
> >> > > >> Thanks.
> >> > > >>
> >> > > >> -Jia
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Sijie Guo <gu...@gmail.com>.
On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
>
>
> 2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
>
>> Thanks a lot for your time Yiming and Enrico. :)
>>
>> Regarding the security, we could do it in a separate BP, and make this BP
>> more focus on filling up the useful endpoints. How about it?
>>
>
> OK, but please consider that the more utility you add to the endpoint the
> more users will want to enable it and so security will be a blocker issue
>
My take on this - we all know security is important for any communication
to bookies.
However, security is a big different scope of problems to address. we
shouldn't put everything in one big BP. I'd suggest us focusing on the
problems
that a BP tends to address, defer other things to separate BPs. Otherwise
we can't land any changes quickly.
>
>
>>
>> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> > Hi Jia,
>> > I am OK with the idea of having standard HTTP API, this will help
>> > development of (non-Java) tools.
>> >
>> > It is not clear to me if we are going to add an http API useful for
>> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper.
>> > I am referring to the fact that in the proposal there are API calls to
>> > create ledgers and to read data.
>> > I think we should separate this two aspects and maybe it is better to
>> > address the 'bookie management' first, which is the work that Yiming
>> > started.
>> >
>> [jia] It is targeting for the admin portal. the existence of the ledger
>> api
>> is just to simplify debugging or operations. we can eliminate the ‘create’
>> endpoint.
>>
>
> OK so are you already thinking about specific tools to manage bookies
> without the bookie shell
>
We would like to integrate this for schedulers (e.g. k8s). so we can deploy
and operate bookkeeper easily
in those environments.
>
>
>>
>>
>> >
>> > Another point very important to me is that if we are going to introduce
>> > management operations via http we have to take into account security, at
>> > least TLS and some kind of authentication.
>> > About TLS it is very simple to achieve this, every HTTP server supports
>> TLS
>> > (https)
>> > About authentication the problem is not so simple, Kerberos on HTTPs is
>> > very complex, but we need to introduce some auth mechanism.
>> > IMHO We can require the 'http server implementation' to implement
>> security
>> > but out of the box we have to supply basic support at least for one
>> > provider bundled in the distribution package.
>> >
>> [jia] The BP focuses on filling up the useful endpoints. The security will
>> be a separate BP.
>>
>>
>> >
>> > Some API can return very large result sets like 'list ledgers', actually
>> > the HTTP Server subsystem exchanges strings in memory, we will need to
>> > introduce some more smart way because it would be easy to bring down the
>> > bookie just by calling that API multiple times concurrencly (it is just
>> an
>> > example)
>> >
>> [jia] We could add pagination into all the `list` api.
>>
>
> Yes but it will be really tricky to implement, but please do it
>
>
>>
>> >
>> > IMHO The 'create ledger' API is not useful, due to the design of BK you
>> > have to create a ledger and then write immediately to it, I think that
>> such
>> > an API should allow the client to stream the contents of the ledger in
>> the
>> > HTTP body at least. But I think that a more stateful http API needs to
>> be
>> > designed to implement a pure Http client
>> >
>> [jia] Thanks, we are not planning to implement an http client. we will
>> remove ‘create’ here.
>>
>
> OK, thanks
>
>>
>>
>> >
>> > Maybe we should add a more narrowed motivation and add the only useful
>> APIs
>> > to address those issues.
>> > For instance in my company we would like to start creating a BookKeeper
>> Web
>> > UI, so we need some way to talk to bookies and exchange data, but in
>> this
>> > case I am interested in asking to bookies their status and the effective
>> > contents of the bookie
>> >
>> [jia] The BP proposes adding a standard naming convention for adding admin
>> endpoints. Feel free to propose the endpoints you would like to appear in
>> the http admin portal.
>>
>
> The ones you wrote are enough complete for me, I would like to have
> read-only operation in order to have a global view on the status of the
> system.
>
> -- Enrico
>
>
>
>>
>>
>> >
>> > -- Enrico
>> >
>> >
>> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
>> >
>> > > Sure, I think the current HTTP endpoints in Twitter are only designed
>> for
>> > > Twitter specific, such as check quorum loss, check rack/region
>> diversity.
>> > > So the endpoints convention in Twitter are not the same as in the
>> > proposal.
>> > > I think it would be great to have an agreement on the API naming
>> design,
>> > so
>> > > I like the API design in the proposal, I think the proposal looks
>> good to
>> > > me.
>> > >
>> > > Besides, we're currently only using GET functionalities in Twitter,
>> but I
>> > > notice there're a lot of POST and PUT APIs in the proposal which could
>> > > change the bookie state or trigger some heavy workload. These APIs
>> looks
>> > a
>> > > bit risky to me if we don't have any authentication enabled (in
>> Twitter).
>> > >
>> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com>
>> wrote:
>> > >
>> > > > + Yiming
>> > > >
>> > > > Yiming, if you have time, please take a look at this BP. let's see
>> if
>> > > > there are any conflicts with those you added for autorecovery.
>> > > >
>> > > > - Sijie
>> > > >
>> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> Based on Github #278 <https://github.com/apache/boo
>> kkeeper/pull/278>,
>> > > I
>> > > >> have just posted a proposal regarding define BookKeeper public http
>> > > >> endpoints:
>> > > >>
>> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
>> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
>> > > >>
>> > > >>
>> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
>> > introduces
>> > > >> BookKeeper Http Endpoint module. However there are only two
>> endpoints,
>> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in
>> #278.
>> > > In
>> > > >> order to fully leverage the http modules, The goal of this BP is to
>> > add
>> > > >> more endpoints to this modules.
>> > > >>
>> > > >> Any comments are welcome and appreciated.
>> > > >>
>> > > >>
>> > > >> Thanks.
>> > > >>
>> > > >> -Jia
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Enrico Olivelli <eo...@gmail.com>.
2017-09-13 10:31 GMT+02:00 Jia Zhai <zh...@gmail.com>:
> Thanks a lot for your time Yiming and Enrico. :)
>
> Regarding the security, we could do it in a separate BP, and make this BP
> more focus on filling up the useful endpoints. How about it?
>
OK, but please consider that the more utility you add to the endpoint the
more users will want to enable it and so security will be a blocker issue
>
> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi Jia,
> > I am OK with the idea of having standard HTTP API, this will help
> > development of (non-Java) tools.
> >
> > It is not clear to me if we are going to add an http API useful for
> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper.
> > I am referring to the fact that in the proposal there are API calls to
> > create ledgers and to read data.
> > I think we should separate this two aspects and maybe it is better to
> > address the 'bookie management' first, which is the work that Yiming
> > started.
> >
> [jia] It is targeting for the admin portal. the existence of the ledger api
> is just to simplify debugging or operations. we can eliminate the ‘create’
> endpoint.
>
OK so are you already thinking about specific tools to manage bookies
without the bookie shell
>
>
> >
> > Another point very important to me is that if we are going to introduce
> > management operations via http we have to take into account security, at
> > least TLS and some kind of authentication.
> > About TLS it is very simple to achieve this, every HTTP server supports
> TLS
> > (https)
> > About authentication the problem is not so simple, Kerberos on HTTPs is
> > very complex, but we need to introduce some auth mechanism.
> > IMHO We can require the 'http server implementation' to implement
> security
> > but out of the box we have to supply basic support at least for one
> > provider bundled in the distribution package.
> >
> [jia] The BP focuses on filling up the useful endpoints. The security will
> be a separate BP.
>
>
> >
> > Some API can return very large result sets like 'list ledgers', actually
> > the HTTP Server subsystem exchanges strings in memory, we will need to
> > introduce some more smart way because it would be easy to bring down the
> > bookie just by calling that API multiple times concurrencly (it is just
> an
> > example)
> >
> [jia] We could add pagination into all the `list` api.
>
Yes but it will be really tricky to implement, but please do it
>
> >
> > IMHO The 'create ledger' API is not useful, due to the design of BK you
> > have to create a ledger and then write immediately to it, I think that
> such
> > an API should allow the client to stream the contents of the ledger in
> the
> > HTTP body at least. But I think that a more stateful http API needs to be
> > designed to implement a pure Http client
> >
> [jia] Thanks, we are not planning to implement an http client. we will
> remove ‘create’ here.
>
OK, thanks
>
>
> >
> > Maybe we should add a more narrowed motivation and add the only useful
> APIs
> > to address those issues.
> > For instance in my company we would like to start creating a BookKeeper
> Web
> > UI, so we need some way to talk to bookies and exchange data, but in this
> > case I am interested in asking to bookies their status and the effective
> > contents of the bookie
> >
> [jia] The BP proposes adding a standard naming convention for adding admin
> endpoints. Feel free to propose the endpoints you would like to appear in
> the http admin portal.
>
The ones you wrote are enough complete for me, I would like to have
read-only operation in order to have a global view on the status of the
system.
-- Enrico
>
>
> >
> > -- Enrico
> >
> >
> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
> >
> > > Sure, I think the current HTTP endpoints in Twitter are only designed
> for
> > > Twitter specific, such as check quorum loss, check rack/region
> diversity.
> > > So the endpoints convention in Twitter are not the same as in the
> > proposal.
> > > I think it would be great to have an agreement on the API naming
> design,
> > so
> > > I like the API design in the proposal, I think the proposal looks good
> to
> > > me.
> > >
> > > Besides, we're currently only using GET functionalities in Twitter,
> but I
> > > notice there're a lot of POST and PUT APIs in the proposal which could
> > > change the bookie state or trigger some heavy workload. These APIs
> looks
> > a
> > > bit risky to me if we don't have any authentication enabled (in
> Twitter).
> > >
> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > + Yiming
> > > >
> > > > Yiming, if you have time, please take a look at this BP. let's see if
> > > > there are any conflicts with those you added for autorecovery.
> > > >
> > > > - Sijie
> > > >
> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com>
> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Based on Github #278 <https://github.com/apache/bookkeeper/pull/278
> >,
> > > I
> > > >> have just posted a proposal regarding define BookKeeper public http
> > > >> endpoints:
> > > >>
> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
> > > >> 17%3A+Define+BookKeeper+public+http+endpoints
> > > >>
> > > >>
> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
> > introduces
> > > >> BookKeeper Http Endpoint module. However there are only two
> endpoints,
> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in
> #278.
> > > In
> > > >> order to fully leverage the http modules, The goal of this BP is to
> > add
> > > >> more endpoints to this modules.
> > > >>
> > > >> Any comments are welcome and appreciated.
> > > >>
> > > >>
> > > >> Thanks.
> > > >>
> > > >> -Jia
> > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Jia Zhai <zh...@gmail.com>.
Thanks a lot for your time Yiming and Enrico. :)
Regarding the security, we could do it in a separate BP, and make this BP
more focus on filling up the useful endpoints. How about it?
On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Hi Jia,
> I am OK with the idea of having standard HTTP API, this will help
> development of (non-Java) tools.
>
> It is not clear to me if we are going to add an http API useful for
> "managing bookies" or a new HTTP REST-like Client API to BookKeeper.
> I am referring to the fact that in the proposal there are API calls to
> create ledgers and to read data.
> I think we should separate this two aspects and maybe it is better to
> address the 'bookie management' first, which is the work that Yiming
> started.
>
[jia] It is targeting for the admin portal. the existence of the ledger api
is just to simplify debugging or operations. we can eliminate the ‘create’
endpoint.
>
> Another point very important to me is that if we are going to introduce
> management operations via http we have to take into account security, at
> least TLS and some kind of authentication.
> About TLS it is very simple to achieve this, every HTTP server supports TLS
> (https)
> About authentication the problem is not so simple, Kerberos on HTTPs is
> very complex, but we need to introduce some auth mechanism.
> IMHO We can require the 'http server implementation' to implement security
> but out of the box we have to supply basic support at least for one
> provider bundled in the distribution package.
>
[jia] The BP focuses on filling up the useful endpoints. The security will
be a separate BP.
>
> Some API can return very large result sets like 'list ledgers', actually
> the HTTP Server subsystem exchanges strings in memory, we will need to
> introduce some more smart way because it would be easy to bring down the
> bookie just by calling that API multiple times concurrencly (it is just an
> example)
>
[jia] We could add pagination into all the `list` api.
>
> IMHO The 'create ledger' API is not useful, due to the design of BK you
> have to create a ledger and then write immediately to it, I think that such
> an API should allow the client to stream the contents of the ledger in the
> HTTP body at least. But I think that a more stateful http API needs to be
> designed to implement a pure Http client
>
[jia] Thanks, we are not planning to implement an http client. we will
remove ‘create’ here.
>
> Maybe we should add a more narrowed motivation and add the only useful APIs
> to address those issues.
> For instance in my company we would like to start creating a BookKeeper Web
> UI, so we need some way to talk to bookies and exchange data, but in this
> case I am interested in asking to bookies their status and the effective
> contents of the bookie
>
[jia] The BP proposes adding a standard naming convention for adding admin
endpoints. Feel free to propose the endpoints you would like to appear in
the http admin portal.
>
> -- Enrico
>
>
> 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
>
> > Sure, I think the current HTTP endpoints in Twitter are only designed for
> > Twitter specific, such as check quorum loss, check rack/region diversity.
> > So the endpoints convention in Twitter are not the same as in the
> proposal.
> > I think it would be great to have an agreement on the API naming design,
> so
> > I like the API design in the proposal, I think the proposal looks good to
> > me.
> >
> > Besides, we're currently only using GET functionalities in Twitter, but I
> > notice there're a lot of POST and PUT APIs in the proposal which could
> > change the bookie state or trigger some heavy workload. These APIs looks
> a
> > bit risky to me if we don't have any authentication enabled (in Twitter).
> >
> > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com> wrote:
> >
> > > + Yiming
> > >
> > > Yiming, if you have time, please take a look at this BP. let's see if
> > > there are any conflicts with those you added for autorecovery.
> > >
> > > - Sijie
> > >
> > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com> wrote:
> > >
> > >> Hi all,
> > >>
> > >> Based on Github #278 <https://github.com/apache/bookkeeper/pull/278>,
> > I
> > >> have just posted a proposal regarding define BookKeeper public http
> > >> endpoints:
> > >>
> > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
> > >> 17%3A+Define+BookKeeper+public+http+endpoints
> > >>
> > >>
> > >> Github #278 <https://github.com/apache/bookkeeper/pull/278>
> introduces
> > >> BookKeeper Http Endpoint module. However there are only two endpoints,
> > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in #278.
> > In
> > >> order to fully leverage the http modules, The goal of this BP is to
> add
> > >> more endpoints to this modules.
> > >>
> > >> Any comments are welcome and appreciated.
> > >>
> > >>
> > >> Thanks.
> > >>
> > >> -Jia
> > >>
> > >
> > >
> >
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Enrico Olivelli <eo...@gmail.com>.
Hi Jia,
I am OK with the idea of having standard HTTP API, this will help
development of (non-Java) tools.
It is not clear to me if we are going to add an http API useful for
"managing bookies" or a new HTTP REST-like Client API to BookKeeper.
I am referring to the fact that in the proposal there are API calls to
create ledgers and to read data.
I think we should separate this two aspects and maybe it is better to
address the 'bookie management' first, which is the work that Yiming
started.
Another point very important to me is that if we are going to introduce
management operations via http we have to take into account security, at
least TLS and some kind of authentication.
About TLS it is very simple to achieve this, every HTTP server supports TLS
(https)
About authentication the problem is not so simple, Kerberos on HTTPs is
very complex, but we need to introduce some auth mechanism.
IMHO We can require the 'http server implementation' to implement security
but out of the box we have to supply basic support at least for one
provider bundled in the distribution package.
Some API can return very large result sets like 'list ledgers', actually
the HTTP Server subsystem exchanges strings in memory, we will need to
introduce some more smart way because it would be easy to bring down the
bookie just by calling that API multiple times concurrencly (it is just an
example)
IMHO The 'create ledger' API is not useful, due to the design of BK you
have to create a ledger and then write immediately to it, I think that such
an API should allow the client to stream the contents of the ledger in the
HTTP body at least. But I think that a more stateful http API needs to be
designed to implement a pure Http client
Maybe we should add a more narrowed motivation and add the only useful APIs
to address those issues.
For instance in my company we would like to start creating a BookKeeper Web
UI, so we need some way to talk to bookies and exchange data, but in this
case I am interested in asking to bookies their status and the effective
contents of the bookie
-- Enrico
2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>:
> Sure, I think the current HTTP endpoints in Twitter are only designed for
> Twitter specific, such as check quorum loss, check rack/region diversity.
> So the endpoints convention in Twitter are not the same as in the proposal.
> I think it would be great to have an agreement on the API naming design, so
> I like the API design in the proposal, I think the proposal looks good to
> me.
>
> Besides, we're currently only using GET functionalities in Twitter, but I
> notice there're a lot of POST and PUT APIs in the proposal which could
> change the bookie state or trigger some heavy workload. These APIs looks a
> bit risky to me if we don't have any authentication enabled (in Twitter).
>
> On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com> wrote:
>
> > + Yiming
> >
> > Yiming, if you have time, please take a look at this BP. let's see if
> > there are any conflicts with those you added for autorecovery.
> >
> > - Sijie
> >
> > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> Based on Github #278 <https://github.com/apache/bookkeeper/pull/278>,
> I
> >> have just posted a proposal regarding define BookKeeper public http
> >> endpoints:
> >>
> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
> >> 17%3A+Define+BookKeeper+public+http+endpoints
> >>
> >>
> >> Github #278 <https://github.com/apache/bookkeeper/pull/278> introduces
> >> BookKeeper Http Endpoint module. However there are only two endpoints,
> >> which is “/heartbeat” and “/api/config/serverconfig”, defined in #278.
> In
> >> order to fully leverage the http modules, The goal of this BP is to add
> >> more endpoints to this modules.
> >>
> >> Any comments are welcome and appreciated.
> >>
> >>
> >> Thanks.
> >>
> >> -Jia
> >>
> >
> >
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Yiming Zang <yz...@twitter.com.INVALID>.
Sure, I think the current HTTP endpoints in Twitter are only designed for
Twitter specific, such as check quorum loss, check rack/region diversity.
So the endpoints convention in Twitter are not the same as in the proposal.
I think it would be great to have an agreement on the API naming design, so
I like the API design in the proposal, I think the proposal looks good to
me.
Besides, we're currently only using GET functionalities in Twitter, but I
notice there're a lot of POST and PUT APIs in the proposal which could
change the bookie state or trigger some heavy workload. These APIs looks a
bit risky to me if we don't have any authentication enabled (in Twitter).
On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <gu...@gmail.com> wrote:
> + Yiming
>
> Yiming, if you have time, please take a look at this BP. let's see if
> there are any conflicts with those you added for autorecovery.
>
> - Sijie
>
> On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com> wrote:
>
>> Hi all,
>>
>> Based on Github #278 <https://github.com/apache/bookkeeper/pull/278>, I
>> have just posted a proposal regarding define BookKeeper public http
>> endpoints:
>>
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-
>> 17%3A+Define+BookKeeper+public+http+endpoints
>>
>>
>> Github #278 <https://github.com/apache/bookkeeper/pull/278> introduces
>> BookKeeper Http Endpoint module. However there are only two endpoints,
>> which is “/heartbeat” and “/api/config/serverconfig”, defined in #278. In
>> order to fully leverage the http modules, The goal of this BP is to add
>> more endpoints to this modules.
>>
>> Any comments are welcome and appreciated.
>>
>>
>> Thanks.
>>
>> -Jia
>>
>
>
Re: [DISCUSS] BP-17: define BookKeeper public http endpoints
Posted by Sijie Guo <gu...@gmail.com>.
+ Yiming
Yiming, if you have time, please take a look at this BP. let's see if there
are any conflicts with those you added for autorecovery.
- Sijie
On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zh...@gmail.com> wrote:
> Hi all,
>
> Based on Github #278 <https://github.com/apache/bookkeeper/pull/278>, I
> have just posted a proposal regarding define BookKeeper public http
> endpoints:
>
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> BP-17%3A+Define+BookKeeper+public+http+endpoints
>
>
> Github #278 <https://github.com/apache/bookkeeper/pull/278> introduces
> BookKeeper Http Endpoint module. However there are only two endpoints,
> which is “/heartbeat” and “/api/config/serverconfig”, defined in #278. In
> order to fully leverage the http modules, The goal of this BP is to add
> more endpoints to this modules.
>
> Any comments are welcome and appreciated.
>
>
> Thanks.
>
> -Jia
>