You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Sijie Guo <si...@apache.org> on 2020/05/13 10:25:06 UTC
[DISCUSS] PIP 64: Introduce REST endpoints for producing, consuming
and reading messages
Hi all,
Some of the people have been asking for a REST-proxy like equivalent
component in Pulsar. Pulsar already has a web service that provides the
management API over a Pulsar cluster. We don't actually need a separate
component or service for supporting streaming events in and out of Pulsar
using HTTP. We can just add endpoints to the existing web server to support
those functionalities.
I'd like to start a discussion on adding REST endpoints to the current web
server for producing, consuming and reading messages. So people can use
HTTP for streaming events in and out of Pulsar without setting up
additional components.
The full proposal is written at
https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
.
Feedback and comments are welcome!
- Sijie
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Sijie Guo <gu...@gmail.com>.
Hi Christophe,
This is to help with applications who are programming over existing web
servers and would like to get the capability to stream events in and out
using HTTP.
- Sijie
On Wed, May 13, 2020 at 11:31 AM Christophe Bornet <bo...@gmail.com>
wrote:
> Hi Sijie,
>
> What will be the difference with the WebSocket proxy ?
>
> Le mer. 13 mai 2020 à 12:25, Sijie Guo <si...@apache.org> a écrit :
>
> > Hi all,
> >
> > Some of the people have been asking for a REST-proxy like equivalent
> > component in Pulsar. Pulsar already has a web service that provides the
> > management API over a Pulsar cluster. We don't actually need a separate
> > component or service for supporting streaming events in and out of Pulsar
> > using HTTP. We can just add endpoints to the existing web server to
> support
> > those functionalities.
> >
> > I'd like to start a discussion on adding REST endpoints to the current
> web
> > server for producing, consuming and reading messages. So people can use
> > HTTP for streaming events in and out of Pulsar without setting up
> > additional components.
> >
> > The full proposal is written at
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > .
> >
> > Feedback and comments are welcome!
> >
> > - Sijie
> >
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Christophe Bornet <bo...@gmail.com>.
Hi Sijie,
What will be the difference with the WebSocket proxy ?
Le mer. 13 mai 2020 à 12:25, Sijie Guo <si...@apache.org> a écrit :
> Hi all,
>
> Some of the people have been asking for a REST-proxy like equivalent
> component in Pulsar. Pulsar already has a web service that provides the
> management API over a Pulsar cluster. We don't actually need a separate
> component or service for supporting streaming events in and out of Pulsar
> using HTTP. We can just add endpoints to the existing web server to support
> those functionalities.
>
> I'd like to start a discussion on adding REST endpoints to the current web
> server for producing, consuming and reading messages. So people can use
> HTTP for streaming events in and out of Pulsar without setting up
> additional components.
>
> The full proposal is written at
>
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> .
>
> Feedback and comments are welcome!
>
> - Sijie
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Shivji Kumar Jha <sh...@gmail.com>.
+1
Feature parity between different pulsar clients will surely benefit from
this. Java client, for instance, is generally ahead of other languages. If
we can have a rest api for streaming, one could just add a feature there
and have all the clients unblocked.
Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512
On Wed, May 13, 2020 at 3:55 PM Sijie Guo <si...@apache.org> wrote:
> Hi all,
>
> Some of the people have been asking for a REST-proxy like equivalent
> component in Pulsar. Pulsar already has a web service that provides the
> management API over a Pulsar cluster. We don't actually need a separate
> component or service for supporting streaming events in and out of Pulsar
> using HTTP. We can just add endpoints to the existing web server to support
> those functionalities.
>
> I'd like to start a discussion on adding REST endpoints to the current web
> server for producing, consuming and reading messages. So people can use
> HTTP for streaming events in and out of Pulsar without setting up
> additional components.
>
> The full proposal is written at
>
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> .
>
> Feedback and comments are welcome!
>
> - Sijie
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Matteo Merli <ma...@gmail.com>.
--
Matteo Merli
<ma...@gmail.com>
On Thu, May 14, 2020 at 12:33 AM Sijie Guo <gu...@gmail.com> wrote:
>
> On Wed, May 13, 2020 at 10:31 PM Matteo Merli <ma...@gmail.com>
> wrote:
> People can use the HOST header for targeting to a specific broker.
>
> If applications don't use the HOST header, it will fall back to the
> redirection approach as what we did for other REST requests. So the request
> goes to the owner broker for consuming and acknowledging messages.
> it works with both brokers and proxies and it is the same as other restful
> requests.
Both reading (and reusing) the host header and following redirections
has several issues and gotchas from the application side. With
hindsight, we shouldn't have relied on redirections for the admin API
either.
The major problems are:
1. HTTP client libs and tools by default don't follow redirects. This
is easily fixable but causes a lot of confusion and troubleshooting
time
2. A service that redirects to a particular host cannot be exposed
through a load balancer. This is very important and has also
ramifications issues on how the TLS certificates need to be created.
3. After a redirect, most HTTP client libs are not forwarding the
Authorization header anymore, for security reasons. In same cases,
it's just the default, while in others (eg: Java default HTTP client)
it's literally not possible to make it pass the header.
Finally, requiring the application to keep the host information
around, would complicate a lot the app logic. Just one example: if I
get an error while deleting a message, do I need to step back and
create a new "session" to read the message again within a new session?
All these requirements are really making punches with the objectives
of REST. In my view, if one wants to have a "session" kind of
behavior, that should rather be done through WebSocket.
> > 3. >>> Note that this request must be made to the specific broker
> > holding the consumer instance. <<<
> > That's not necessary to go on the same host. A shared
> > subscription dispatcher will be able to apply the ack even if it's
> > received from a different consumer (and it will clean up the pending
> > acks).
> >
>
> It is yes for shared subscriptions. But it doesn't work for other types of
> subscriptions.
> We would like to start with this approach to make it work seamlessly for
> different types of subscriptions.
> We can look into how to optimize it as it goes.
I believe that a simple approach that works with no special
requirements for a subset of features is 10x better than trying to
expose every single feature but, in the process, making the entire API
very convoluted and difficult to use.
This is even more true if:
1. The stated goal for this REST API is to offer "easy" access
2. There's already an implementation that is better suited for that
kind of task (eg: WebSocket).
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Sijie Guo <gu...@gmail.com>.
Hi Joe,
Thank you for your feedback! I don't think this is going to touch existing
protobuf based throughput. But we will do what you have suggested.
- Sijie
On Thu, May 14, 2020 at 6:49 AM Joe Francis <jo...@verizonmedia.com.invalid>
wrote:
> We do have concerns, and we have implemented throttles on even our existing
> REST APIs in the broker.
> So we would like to be able to turn this off, and validation that the
> implementation changes do not degrade existing protobuf based throughput
>
> Joe
>
> On Thu, May 14, 2020 at 12:35 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > On Wed, May 13, 2020 at 10:31 PM Matteo Merli <ma...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > The REST API is more simple and intuitive for developers compared to
> > > WebSocket.
> > >
> > > While it's easy to get started with Websocket API, the biggest problem
> > > for applications is how to handle errors and corner cases. Eg:
> > > re-establishing a producer session while maintaining a queue of
> > > pending messages. This is overly complicated and that's what the
> > > regular Pulsar client lib already solves.
> > >
> > > Couple of comments on the proposal:
> > > 1. I'd like to see the option, for example based on different
> > > Content-Types to have simplified body. eg: `curl -XPOST $URL -d
> > > @file.txt` could be able to just publish a raw bytes payload on the
> > > given topic.
> > >
> >
> > We can keep the option available.
> >
> >
> > > 2. >>> Because consumers are stateful, any consumer created with the
> > > REST API are tied to a specific broker. <<<
> > > I'd avoid this requirement since it makes app devs life much more
> > > complicated.
> > >
> >
> > People can use the HOST header for targeting to a specific broker.
> >
> > If applications don't use the HOST header, it will fall back to the
> > redirection approach as what we did for other REST requests. So the
> request
> > goes to the owner broker for consuming and acknowledging messages.
> > it works with both brokers and proxies and it is the same as other
> restful
> > requests.
> >
> >
> >
> > > 3. >>> Note that this request must be made to the specific broker
> > > holding the consumer instance. <<<
> > > That's not necessary to go on the same host. A shared
> > > subscription dispatcher will be able to apply the ack even if it's
> > > received from a different consumer (and it will clean up the pending
> > > acks).
> > >
> >
> > It is yes for shared subscriptions. But it doesn't work for other types
> of
> > subscriptions.
> > We would like to start with this approach to make it work seamlessly for
> > different types of subscriptions.
> > We can look into how to optimize it as it goes.
> >
> >
> > >
> > > --
> > > Matteo Merli
> > > <ma...@gmail.com>
> > >
> > > On Wed, May 13, 2020 at 6:19 PM Jia Zhai <zh...@gmail.com> wrote:
> > > >
> > > > +1
> > > >
> > > > Best Regards.
> > > >
> > > >
> > > > Jia Zhai
> > > >
> > > > Beijing, China
> > > >
> > > > Mobile: +86 15810491983
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 13, 2020 at 6:25 PM Sijie Guo <si...@apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Some of the people have been asking for a REST-proxy like
> equivalent
> > > > > component in Pulsar. Pulsar already has a web service that provides
> > the
> > > > > management API over a Pulsar cluster. We don't actually need a
> > separate
> > > > > component or service for supporting streaming events in and out of
> > > Pulsar
> > > > > using HTTP. We can just add endpoints to the existing web server to
> > > support
> > > > > those functionalities.
> > > > >
> > > > > I'd like to start a discussion on adding REST endpoints to the
> > current
> > > web
> > > > > server for producing, consuming and reading messages. So people can
> > use
> > > > > HTTP for streaming events in and out of Pulsar without setting up
> > > > > additional components.
> > > > >
> > > > > The full proposal is written at
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > > > > .
> > > > >
> > > > > Feedback and comments are welcome!
> > > > >
> > > > > - Sijie
> > > > >
> > >
> >
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Joe Francis <jo...@verizonmedia.com.INVALID>.
We do have concerns, and we have implemented throttles on even our existing
REST APIs in the broker.
So we would like to be able to turn this off, and validation that the
implementation changes do not degrade existing protobuf based throughput
Joe
On Thu, May 14, 2020 at 12:35 AM Sijie Guo <gu...@gmail.com> wrote:
> On Wed, May 13, 2020 at 10:31 PM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > +1
> >
> > The REST API is more simple and intuitive for developers compared to
> > WebSocket.
> >
> > While it's easy to get started with Websocket API, the biggest problem
> > for applications is how to handle errors and corner cases. Eg:
> > re-establishing a producer session while maintaining a queue of
> > pending messages. This is overly complicated and that's what the
> > regular Pulsar client lib already solves.
> >
> > Couple of comments on the proposal:
> > 1. I'd like to see the option, for example based on different
> > Content-Types to have simplified body. eg: `curl -XPOST $URL -d
> > @file.txt` could be able to just publish a raw bytes payload on the
> > given topic.
> >
>
> We can keep the option available.
>
>
> > 2. >>> Because consumers are stateful, any consumer created with the
> > REST API are tied to a specific broker. <<<
> > I'd avoid this requirement since it makes app devs life much more
> > complicated.
> >
>
> People can use the HOST header for targeting to a specific broker.
>
> If applications don't use the HOST header, it will fall back to the
> redirection approach as what we did for other REST requests. So the request
> goes to the owner broker for consuming and acknowledging messages.
> it works with both brokers and proxies and it is the same as other restful
> requests.
>
>
>
> > 3. >>> Note that this request must be made to the specific broker
> > holding the consumer instance. <<<
> > That's not necessary to go on the same host. A shared
> > subscription dispatcher will be able to apply the ack even if it's
> > received from a different consumer (and it will clean up the pending
> > acks).
> >
>
> It is yes for shared subscriptions. But it doesn't work for other types of
> subscriptions.
> We would like to start with this approach to make it work seamlessly for
> different types of subscriptions.
> We can look into how to optimize it as it goes.
>
>
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> > On Wed, May 13, 2020 at 6:19 PM Jia Zhai <zh...@gmail.com> wrote:
> > >
> > > +1
> > >
> > > Best Regards.
> > >
> > >
> > > Jia Zhai
> > >
> > > Beijing, China
> > >
> > > Mobile: +86 15810491983
> > >
> > >
> > >
> > >
> > > On Wed, May 13, 2020 at 6:25 PM Sijie Guo <si...@apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Some of the people have been asking for a REST-proxy like equivalent
> > > > component in Pulsar. Pulsar already has a web service that provides
> the
> > > > management API over a Pulsar cluster. We don't actually need a
> separate
> > > > component or service for supporting streaming events in and out of
> > Pulsar
> > > > using HTTP. We can just add endpoints to the existing web server to
> > support
> > > > those functionalities.
> > > >
> > > > I'd like to start a discussion on adding REST endpoints to the
> current
> > web
> > > > server for producing, consuming and reading messages. So people can
> use
> > > > HTTP for streaming events in and out of Pulsar without setting up
> > > > additional components.
> > > >
> > > > The full proposal is written at
> > > >
> > > >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > > > .
> > > >
> > > > Feedback and comments are welcome!
> > > >
> > > > - Sijie
> > > >
> >
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Sijie Guo <gu...@gmail.com>.
On Wed, May 13, 2020 at 10:31 PM Matteo Merli <ma...@gmail.com>
wrote:
> +1
>
> The REST API is more simple and intuitive for developers compared to
> WebSocket.
>
> While it's easy to get started with Websocket API, the biggest problem
> for applications is how to handle errors and corner cases. Eg:
> re-establishing a producer session while maintaining a queue of
> pending messages. This is overly complicated and that's what the
> regular Pulsar client lib already solves.
>
> Couple of comments on the proposal:
> 1. I'd like to see the option, for example based on different
> Content-Types to have simplified body. eg: `curl -XPOST $URL -d
> @file.txt` could be able to just publish a raw bytes payload on the
> given topic.
>
We can keep the option available.
> 2. >>> Because consumers are stateful, any consumer created with the
> REST API are tied to a specific broker. <<<
> I'd avoid this requirement since it makes app devs life much more
> complicated.
>
People can use the HOST header for targeting to a specific broker.
If applications don't use the HOST header, it will fall back to the
redirection approach as what we did for other REST requests. So the request
goes to the owner broker for consuming and acknowledging messages.
it works with both brokers and proxies and it is the same as other restful
requests.
> 3. >>> Note that this request must be made to the specific broker
> holding the consumer instance. <<<
> That's not necessary to go on the same host. A shared
> subscription dispatcher will be able to apply the ack even if it's
> received from a different consumer (and it will clean up the pending
> acks).
>
It is yes for shared subscriptions. But it doesn't work for other types of
subscriptions.
We would like to start with this approach to make it work seamlessly for
different types of subscriptions.
We can look into how to optimize it as it goes.
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Wed, May 13, 2020 at 6:19 PM Jia Zhai <zh...@gmail.com> wrote:
> >
> > +1
> >
> > Best Regards.
> >
> >
> > Jia Zhai
> >
> > Beijing, China
> >
> > Mobile: +86 15810491983
> >
> >
> >
> >
> > On Wed, May 13, 2020 at 6:25 PM Sijie Guo <si...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Some of the people have been asking for a REST-proxy like equivalent
> > > component in Pulsar. Pulsar already has a web service that provides the
> > > management API over a Pulsar cluster. We don't actually need a separate
> > > component or service for supporting streaming events in and out of
> Pulsar
> > > using HTTP. We can just add endpoints to the existing web server to
> support
> > > those functionalities.
> > >
> > > I'd like to start a discussion on adding REST endpoints to the current
> web
> > > server for producing, consuming and reading messages. So people can use
> > > HTTP for streaming events in and out of Pulsar without setting up
> > > additional components.
> > >
> > > The full proposal is written at
> > >
> > >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > > .
> > >
> > > Feedback and comments are welcome!
> > >
> > > - Sijie
> > >
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Matteo Merli <ma...@gmail.com>.
+1
The REST API is more simple and intuitive for developers compared to WebSocket.
While it's easy to get started with Websocket API, the biggest problem
for applications is how to handle errors and corner cases. Eg:
re-establishing a producer session while maintaining a queue of
pending messages. This is overly complicated and that's what the
regular Pulsar client lib already solves.
Couple of comments on the proposal:
1. I'd like to see the option, for example based on different
Content-Types to have simplified body. eg: `curl -XPOST $URL -d
@file.txt` could be able to just publish a raw bytes payload on the
given topic.
2. >>> Because consumers are stateful, any consumer created with the
REST API are tied to a specific broker. <<<
I'd avoid this requirement since it makes app devs life much more
complicated.
3. >>> Note that this request must be made to the specific broker
holding the consumer instance. <<<
That's not necessary to go on the same host. A shared
subscription dispatcher will be able to apply the ack even if it's
received from a different consumer (and it will clean up the pending
acks).
--
Matteo Merli
<ma...@gmail.com>
On Wed, May 13, 2020 at 6:19 PM Jia Zhai <zh...@gmail.com> wrote:
>
> +1
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Wed, May 13, 2020 at 6:25 PM Sijie Guo <si...@apache.org> wrote:
>
> > Hi all,
> >
> > Some of the people have been asking for a REST-proxy like equivalent
> > component in Pulsar. Pulsar already has a web service that provides the
> > management API over a Pulsar cluster. We don't actually need a separate
> > component or service for supporting streaming events in and out of Pulsar
> > using HTTP. We can just add endpoints to the existing web server to support
> > those functionalities.
> >
> > I'd like to start a discussion on adding REST endpoints to the current web
> > server for producing, consuming and reading messages. So people can use
> > HTTP for streaming events in and out of Pulsar without setting up
> > additional components.
> >
> > The full proposal is written at
> >
> > https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > .
> >
> > Feedback and comments are welcome!
> >
> > - Sijie
> >
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Jia Zhai <zh...@gmail.com>.
+1
Best Regards.
Jia Zhai
Beijing, China
Mobile: +86 15810491983
On Wed, May 13, 2020 at 6:25 PM Sijie Guo <si...@apache.org> wrote:
> Hi all,
>
> Some of the people have been asking for a REST-proxy like equivalent
> component in Pulsar. Pulsar already has a web service that provides the
> management API over a Pulsar cluster. We don't actually need a separate
> component or service for supporting streaming events in and out of Pulsar
> using HTTP. We can just add endpoints to the existing web server to support
> those functionalities.
>
> I'd like to start a discussion on adding REST endpoints to the current web
> server for producing, consuming and reading messages. So people can use
> HTTP for streaming events in and out of Pulsar without setting up
> additional components.
>
> The full proposal is written at
>
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> .
>
> Feedback and comments are welcome!
>
> - Sijie
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Sijie Guo <gu...@gmail.com>.
Hi Ming,
Please check my comment above. The reason we don't add it as a separate
service is that Pulsar already has an HTTP server and a lot of applications
are already programming over it. Adding another component or a separate
service that is overlapped with existing Pulsar functionalities doesn't
resolve the problem. Instead, it confuses Pulsar users. We are trying to
provide Pulsar users with a seamless integration experience rather than
installing many different services and components.
- Sijie
On Wed, May 13, 2020 at 3:36 PM Ming Luo <mi...@kafkaesque.io.invalid>
wrote:
> Hi,
>
> Thanks for Sijie's proposal. I resonate Rajan's comment that this project
> should become a peripheral to Pulsar because of the stateless nature of
> HTTP and scaling up a web service inside Pulsar might be affecting
> its current design. For that reason, I developed a project with an HTTP
> interface to Pulsar and webhook notification that pushes messages to
> consumers.
>
> It is written in Go. The current repo is at
> https://github.com/kafkaesque-io/pulsar-beam It can be deployed in
> standalone or inside a Pulsar cluster. I know someone has started using it
> in their production trial. It makes sense for the HTTP interface to
> become an optional add-on to Pulsar. I would like to contribute this back
> to Apache Software Foundation as a standalone repo like Pulsar Manager.
> What do you think? Let me know how I can help.
>
> Cheers
> Ming
>
> On Wed, 13 May 2020 at 17:00, Rajan Dhabalia <rd...@apache.org> wrote:
>
> > *Hi,I would like to add little history about Websocket vs REST API
> > implementation in Pulsar. Before open-sourcing Pulsar and WebSocket proxy
> > in Yahoo, we had implemented and evaluated both WebSocket and REST-HTTP
> > APIs options and then decided to not introduce REST-API and kept
> Websocket
> > proxy solution in Pulsar. Let me add a few points to understand why we
> > should avoid adding REST API in Pulsar. 1. One of the advantages of REST
> > architecture is statelessness which means that any server can handle any
> > request. However, it’s true for Websocket protocol as well but Websoket
> is
> > a bi-directional protocol with full duplex communication and
> client-server
> > can communicate on a single TCP connection which avoids different
> overheads
> > such as authentication on every request that needs to be done on
> REST-Http
> > api.2. If client doesn’t use persist connection correctly on REST-HTTP
> then
> > every produce/consume request can go to a different server and it will
> > create a lot of complexity to manage the control flow, producer/consumer
> > queuing at REST-server which internally manages pulsar-client, clean up
> > unused resources, etc.3. Performance and scalability: Websocket is highly
> > scalable and provides higher performance and throughput compare to
> > REST-HTTP.4. Almost all programming languages and web-browsers
> > <https://caniuse.com/#feat=websockets> support WebSocket.5. Pulsar is
> > known
> > for performance and scalability, so, we try to select solutions that work
> > with scale and provide better performance.6. Pulsar broker gives an
> option
> > to start WebSocket proxy along with broker and that doesn’t require extra
> > component to manage. So, if one doesn’t want to manage additional
> component
> > then one can start with the broker process as well. 7. Other messaging
> > system also moving away from their legacy REST-HTTP api to WebSocket
> > because of performance and scalability concerns.*
> > *So, I think let’s not add this feature in Pulsar but it can be added a
> > utility project outside of pulsar if it is needed for specific usecases.*
> >
> > *Thanks,*
> >
> > *Rajan*
> >
> > On Wed, May 13, 2020 at 3:25 AM Sijie Guo <si...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Some of the people have been asking for a REST-proxy like equivalent
> > > component in Pulsar. Pulsar already has a web service that provides the
> > > management API over a Pulsar cluster. We don't actually need a separate
> > > component or service for supporting streaming events in and out of
> Pulsar
> > > using HTTP. We can just add endpoints to the existing web server to
> > support
> > > those functionalities.
> > >
> > > I'd like to start a discussion on adding REST endpoints to the current
> > web
> > > server for producing, consuming and reading messages. So people can use
> > > HTTP for streaming events in and out of Pulsar without setting up
> > > additional components.
> > >
> > > The full proposal is written at
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > > .
> > >
> > > Feedback and comments are welcome!
> > >
> > > - Sijie
> > >
> >
>
>
> --
> Powered by Pulsar
> Engineering, Kafkaesque
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Ming Luo <mi...@kafkaesque.io.INVALID>.
Hi,
Thanks for Sijie's proposal. I resonate Rajan's comment that this project
should become a peripheral to Pulsar because of the stateless nature of
HTTP and scaling up a web service inside Pulsar might be affecting
its current design. For that reason, I developed a project with an HTTP
interface to Pulsar and webhook notification that pushes messages to
consumers.
It is written in Go. The current repo is at
https://github.com/kafkaesque-io/pulsar-beam It can be deployed in
standalone or inside a Pulsar cluster. I know someone has started using it
in their production trial. It makes sense for the HTTP interface to
become an optional add-on to Pulsar. I would like to contribute this back
to Apache Software Foundation as a standalone repo like Pulsar Manager.
What do you think? Let me know how I can help.
Cheers
Ming
On Wed, 13 May 2020 at 17:00, Rajan Dhabalia <rd...@apache.org> wrote:
> *Hi,I would like to add little history about Websocket vs REST API
> implementation in Pulsar. Before open-sourcing Pulsar and WebSocket proxy
> in Yahoo, we had implemented and evaluated both WebSocket and REST-HTTP
> APIs options and then decided to not introduce REST-API and kept Websocket
> proxy solution in Pulsar. Let me add a few points to understand why we
> should avoid adding REST API in Pulsar. 1. One of the advantages of REST
> architecture is statelessness which means that any server can handle any
> request. However, it’s true for Websocket protocol as well but Websoket is
> a bi-directional protocol with full duplex communication and client-server
> can communicate on a single TCP connection which avoids different overheads
> such as authentication on every request that needs to be done on REST-Http
> api.2. If client doesn’t use persist connection correctly on REST-HTTP then
> every produce/consume request can go to a different server and it will
> create a lot of complexity to manage the control flow, producer/consumer
> queuing at REST-server which internally manages pulsar-client, clean up
> unused resources, etc.3. Performance and scalability: Websocket is highly
> scalable and provides higher performance and throughput compare to
> REST-HTTP.4. Almost all programming languages and web-browsers
> <https://caniuse.com/#feat=websockets> support WebSocket.5. Pulsar is
> known
> for performance and scalability, so, we try to select solutions that work
> with scale and provide better performance.6. Pulsar broker gives an option
> to start WebSocket proxy along with broker and that doesn’t require extra
> component to manage. So, if one doesn’t want to manage additional component
> then one can start with the broker process as well. 7. Other messaging
> system also moving away from their legacy REST-HTTP api to WebSocket
> because of performance and scalability concerns.*
> *So, I think let’s not add this feature in Pulsar but it can be added a
> utility project outside of pulsar if it is needed for specific usecases.*
>
> *Thanks,*
>
> *Rajan*
>
> On Wed, May 13, 2020 at 3:25 AM Sijie Guo <si...@apache.org> wrote:
>
> > Hi all,
> >
> > Some of the people have been asking for a REST-proxy like equivalent
> > component in Pulsar. Pulsar already has a web service that provides the
> > management API over a Pulsar cluster. We don't actually need a separate
> > component or service for supporting streaming events in and out of Pulsar
> > using HTTP. We can just add endpoints to the existing web server to
> support
> > those functionalities.
> >
> > I'd like to start a discussion on adding REST endpoints to the current
> web
> > server for producing, consuming and reading messages. So people can use
> > HTTP for streaming events in and out of Pulsar without setting up
> > additional components.
> >
> > The full proposal is written at
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > .
> >
> > Feedback and comments are welcome!
> >
> > - Sijie
> >
>
--
Powered by Pulsar
Engineering, Kafkaesque
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Sijie Guo <gu...@gmail.com>.
Hi Rajan,
Thank you for your feedback! I understand that Websocket has a lot of
advantages over HTTP.
HTTP is much simple compared to WebSocket. Especially for people who
already program using current admin API,
it makes sense for those applications pointing to one web service rather
than a different util project which can become very complicated.
We are adding those endpoints to separate endpoints. So it doesn't change
any existing endpoints. Does that make sense to you?
- Sijie
On Wed, May 13, 2020 at 2:00 PM Rajan Dhabalia <rd...@apache.org> wrote:
> *Hi,I would like to add little history about Websocket vs REST API
> implementation in Pulsar. Before open-sourcing Pulsar and WebSocket proxy
> in Yahoo, we had implemented and evaluated both WebSocket and REST-HTTP
> APIs options and then decided to not introduce REST-API and kept Websocket
> proxy solution in Pulsar. Let me add a few points to understand why we
> should avoid adding REST API in Pulsar. 1. One of the advantages of REST
> architecture is statelessness which means that any server can handle any
> request. However, it’s true for Websocket protocol as well but Websoket is
> a bi-directional protocol with full duplex communication and client-server
> can communicate on a single TCP connection which avoids different overheads
> such as authentication on every request that needs to be done on REST-Http
> api.2. If client doesn’t use persist connection correctly on REST-HTTP then
> every produce/consume request can go to a different server and it will
> create a lot of complexity to manage the control flow, producer/consumer
> queuing at REST-server which internally manages pulsar-client, clean up
> unused resources, etc.3. Performance and scalability: Websocket is highly
> scalable and provides higher performance and throughput compare to
> REST-HTTP.4. Almost all programming languages and web-browsers
> <https://caniuse.com/#feat=websockets> support WebSocket.5. Pulsar is
> known
> for performance and scalability, so, we try to select solutions that work
> with scale and provide better performance.6. Pulsar broker gives an option
> to start WebSocket proxy along with broker and that doesn’t require extra
> component to manage. So, if one doesn’t want to manage additional component
> then one can start with the broker process as well. 7. Other messaging
> system also moving away from their legacy REST-HTTP api to WebSocket
> because of performance and scalability concerns.*
> *So, I think let’s not add this feature in Pulsar but it can be added a
> utility project outside of pulsar if it is needed for specific usecases.*
>
> *Thanks,*
>
> *Rajan*
>
> On Wed, May 13, 2020 at 3:25 AM Sijie Guo <si...@apache.org> wrote:
>
> > Hi all,
> >
> > Some of the people have been asking for a REST-proxy like equivalent
> > component in Pulsar. Pulsar already has a web service that provides the
> > management API over a Pulsar cluster. We don't actually need a separate
> > component or service for supporting streaming events in and out of Pulsar
> > using HTTP. We can just add endpoints to the existing web server to
> support
> > those functionalities.
> >
> > I'd like to start a discussion on adding REST endpoints to the current
> web
> > server for producing, consuming and reading messages. So people can use
> > HTTP for streaming events in and out of Pulsar without setting up
> > additional components.
> >
> > The full proposal is written at
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> > .
> >
> > Feedback and comments are welcome!
> >
> > - Sijie
> >
>
Re: [DISCUSS] PIP 64: Introduce REST endpoints for producing,
consuming and reading messages
Posted by Rajan Dhabalia <rd...@apache.org>.
*Hi,I would like to add little history about Websocket vs REST API
implementation in Pulsar. Before open-sourcing Pulsar and WebSocket proxy
in Yahoo, we had implemented and evaluated both WebSocket and REST-HTTP
APIs options and then decided to not introduce REST-API and kept Websocket
proxy solution in Pulsar. Let me add a few points to understand why we
should avoid adding REST API in Pulsar. 1. One of the advantages of REST
architecture is statelessness which means that any server can handle any
request. However, it’s true for Websocket protocol as well but Websoket is
a bi-directional protocol with full duplex communication and client-server
can communicate on a single TCP connection which avoids different overheads
such as authentication on every request that needs to be done on REST-Http
api.2. If client doesn’t use persist connection correctly on REST-HTTP then
every produce/consume request can go to a different server and it will
create a lot of complexity to manage the control flow, producer/consumer
queuing at REST-server which internally manages pulsar-client, clean up
unused resources, etc.3. Performance and scalability: Websocket is highly
scalable and provides higher performance and throughput compare to
REST-HTTP.4. Almost all programming languages and web-browsers
<https://caniuse.com/#feat=websockets> support WebSocket.5. Pulsar is known
for performance and scalability, so, we try to select solutions that work
with scale and provide better performance.6. Pulsar broker gives an option
to start WebSocket proxy along with broker and that doesn’t require extra
component to manage. So, if one doesn’t want to manage additional component
then one can start with the broker process as well. 7. Other messaging
system also moving away from their legacy REST-HTTP api to WebSocket
because of performance and scalability concerns.*
*So, I think let’s not add this feature in Pulsar but it can be added a
utility project outside of pulsar if it is needed for specific usecases.*
*Thanks,*
*Rajan*
On Wed, May 13, 2020 at 3:25 AM Sijie Guo <si...@apache.org> wrote:
> Hi all,
>
> Some of the people have been asking for a REST-proxy like equivalent
> component in Pulsar. Pulsar already has a web service that provides the
> management API over a Pulsar cluster. We don't actually need a separate
> component or service for supporting streaming events in and out of Pulsar
> using HTTP. We can just add endpoints to the existing web server to support
> those functionalities.
>
> I'd like to start a discussion on adding REST endpoints to the current web
> server for producing, consuming and reading messages. So people can use
> HTTP for streaming events in and out of Pulsar without setting up
> additional components.
>
> The full proposal is written at
>
> https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages
> .
>
> Feedback and comments are welcome!
>
> - Sijie
>