You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by mattison chao <ma...@apache.org> on 2022/06/06 13:33:16 UTC
[DISCUSS] Implementation for HTTP endpoint producer/consumer
Hi, Pulsar Community,
We have the PIP-64 that introduces HTTP Rest API for producing/consuming
messages(
https://github.com/apache/pulsar/wiki/PIP-64%3A-Introduce-REST-endpoints-for-producing%2C-consuming-and-reading-messages
). But this proposal does not define the implementation.
However, we already have producer HTTP API at the broker side. But, there
are some problems, so refactored in this patch:
https://github.com/apache/pulsar/pull/15876.
Then we add HTTP consumer in this patch:
https://github.com/apache/pulsar/pull/15942.
But, currently have some ideas that do not reach a consensus. Like @Lari
Hotaril mentioned at pull request
https://github.com/apache/pulsar/pull/15942.
It might not be a good idea to add the implementation to the main Pulsar
Admin API at all.
HTTP consuming would be better to handle in a separate component. PIP-64
doesn't determine that this should be part of Pulsar Admin API and we
should revisit this decision. I think it's a bad idea to add HTTP consuming
to Pulsar Admin API and brokers.
I want to discuss whether we should implement the HTTP endpoint in the
broker or separate it at another component(like pulsar-WebSocket).
Best,
Mattison
Re: [DISCUSS] Implementation for HTTP endpoint producer/consumer
Posted by mattison chao <ma...@apache.org>.
>
> If this is a REST endpoint then call it REST. It is very likely that users
> will want to use HTTPS to use it. Calling it HTTP is a misnomer.
Very thanks for your correction.
Best,
Mattison
On Thu, 9 Jun 2022 at 15:37, mattison chao <ma...@apache.org> wrote:
> I think building a separate HTTP server to serve REST produce/consume
>> requests might be a good idea, like FunctionWorkerService, users can
>> choose
>> to run with broker for simplicity or run as a separate component if user
>> wants isolation and scale independently.
>
>
> Yes, it's a great suggestion. We will follow this way to make it more
> scalable.
>
> I would make a separate project and release it as a .nar. It can run as a
>> Broker Protocol Handler or a Proxy Extension.
>
>
> Then if the project gets traction we can add it to Pulsar core repo.
>
>
> +1, We chose to make it a separate project to support the Proxy/Broker
> additional servlet and support it to run as an independent server.
>
> Best,
> Mattison
>
Re: [DISCUSS] Implementation for HTTP endpoint producer/consumer
Posted by mattison chao <ma...@apache.org>.
>
> I think building a separate HTTP server to serve REST produce/consume
> requests might be a good idea, like FunctionWorkerService, users can choose
> to run with broker for simplicity or run as a separate component if user
> wants isolation and scale independently.
Yes, it's a great suggestion. We will follow this way to make it more
scalable.
I would make a separate project and release it as a .nar. It can run as a
> Broker Protocol Handler or a Proxy Extension.
Then if the project gets traction we can add it to Pulsar core repo.
+1, We chose to make it a separate project to support the Proxy/Broker
additional servlet and support it to run as an independent server.
Best,
Mattison
Re: [DISCUSS] Implementation for HTTP endpoint producer/consumer
Posted by Enrico Olivelli <eo...@gmail.com>.
I would make a separate project and release it as a .nar. It can run as a
Broker Protocol Handler or a Proxy Extension.
Then if the project gets traction we can add it to Pulsar core repo.
Enrico and
Il Mar 7 Giu 2022, 17:05 Dave Fisher <wa...@apache.org> ha scritto:
> If this is a REST endpoint then call it REST. It is very likely that users
> will want to use HTTPS to use it. Calling it HTTP is a misnomer.
>
> All The Best,
> Dave
>
> Sent from my iPhone
>
> > On Jun 7, 2022, at 7:25 AM, Zhengxin Cai <ca...@gmail.com> wrote:
> >
> > Thanks for bringing this up.
> > I think building a separate HTTP server to serve REST produce/consume
> > requests might be a good idea, like FunctionWorkerService, users can
> choose
> > to run with broker for simplicity or run as a separate component if user
> > wants isolation and scale independently.
> > I think we just missed this option when building V1, I think it's working
> > considering.
> >
> > mattison chao <ma...@apache.org> 于2022年6月6日周一 21:33写道：
> >
> >> Hi, Pulsar Community,
> >>
> >> We have the PIP-64 that introduces HTTP Rest API for producing/consuming
> >> messages(
> >>
> >>
> https://github.com/apache/pulsar/wiki/PIP-64%3A-Introduce-REST-endpoints-for-producing%2C-consuming-and-reading-messages
> >> ). But this proposal does not define the implementation.
> >>
> >> However, we already have producer HTTP API at the broker side. But,
> there
> >> are some problems, so refactored in this patch:
> >> https://github.com/apache/pulsar/pull/15876.
> >>
> >> Then we add HTTP consumer in this patch:
> >> https://github.com/apache/pulsar/pull/15942.
> >>
> >> But, currently have some ideas that do not reach a consensus. Like @Lari
> >> Hotaril mentioned at pull request
> >> https://github.com/apache/pulsar/pull/15942.
> >>
> >> It might not be a good idea to add the implementation to the main Pulsar
> >> Admin API at all.
> >>
> >> HTTP consuming would be better to handle in a separate component. PIP-64
> >> doesn't determine that this should be part of Pulsar Admin API and we
> >> should revisit this decision. I think it's a bad idea to add HTTP
> consuming
> >> to Pulsar Admin API and brokers.
> >>
> >> I want to discuss whether we should implement the HTTP endpoint in the
> >> broker or separate it at another component(like pulsar-WebSocket).
> >>
> >> Best,
> >>
> >> Mattison
> >>
>
>
Re: [DISCUSS] Implementation for HTTP endpoint producer/consumer
Posted by Dave Fisher <wa...@apache.org>.
If this is a REST endpoint then call it REST. It is very likely that users will want to use HTTPS to use it. Calling it HTTP is a misnomer.
All The Best,
Dave
Sent from my iPhone
> On Jun 7, 2022, at 7:25 AM, Zhengxin Cai <ca...@gmail.com> wrote:
>
> Thanks for bringing this up.
> I think building a separate HTTP server to serve REST produce/consume
> requests might be a good idea, like FunctionWorkerService, users can choose
> to run with broker for simplicity or run as a separate component if user
> wants isolation and scale independently.
> I think we just missed this option when building V1, I think it's working
> considering.
>
> mattison chao <ma...@apache.org> 于2022年6月6日周一 21:33写道：
>
>> Hi, Pulsar Community,
>>
>> We have the PIP-64 that introduces HTTP Rest API for producing/consuming
>> messages(
>>
>> https://github.com/apache/pulsar/wiki/PIP-64%3A-Introduce-REST-endpoints-for-producing%2C-consuming-and-reading-messages
>> ). But this proposal does not define the implementation.
>>
>> However, we already have producer HTTP API at the broker side. But, there
>> are some problems, so refactored in this patch:
>> https://github.com/apache/pulsar/pull/15876.
>>
>> Then we add HTTP consumer in this patch:
>> https://github.com/apache/pulsar/pull/15942.
>>
>> But, currently have some ideas that do not reach a consensus. Like @Lari
>> Hotaril mentioned at pull request
>> https://github.com/apache/pulsar/pull/15942.
>>
>> It might not be a good idea to add the implementation to the main Pulsar
>> Admin API at all.
>>
>> HTTP consuming would be better to handle in a separate component. PIP-64
>> doesn't determine that this should be part of Pulsar Admin API and we
>> should revisit this decision. I think it's a bad idea to add HTTP consuming
>> to Pulsar Admin API and brokers.
>>
>> I want to discuss whether we should implement the HTTP endpoint in the
>> broker or separate it at another component(like pulsar-WebSocket).
>>
>> Best,
>>
>> Mattison
>>
Re: [DISCUSS] Implementation for HTTP endpoint producer/consumer
Posted by Zhengxin Cai <ca...@gmail.com>.
Thanks for bringing this up.
I think building a separate HTTP server to serve REST produce/consume
requests might be a good idea, like FunctionWorkerService, users can choose
to run with broker for simplicity or run as a separate component if user
wants isolation and scale independently.
I think we just missed this option when building V1, I think it's working
considering.
mattison chao <ma...@apache.org> 于2022年6月6日周一 21:33写道：
> Hi, Pulsar Community,
>
> We have the PIP-64 that introduces HTTP Rest API for producing/consuming
> messages(
>
> https://github.com/apache/pulsar/wiki/PIP-64%3A-Introduce-REST-endpoints-for-producing%2C-consuming-and-reading-messages
> ). But this proposal does not define the implementation.
>
> However, we already have producer HTTP API at the broker side. But, there
> are some problems, so refactored in this patch:
> https://github.com/apache/pulsar/pull/15876.
>
> Then we add HTTP consumer in this patch:
> https://github.com/apache/pulsar/pull/15942.
>
> But, currently have some ideas that do not reach a consensus. Like @Lari
> Hotaril mentioned at pull request
> https://github.com/apache/pulsar/pull/15942.
>
> It might not be a good idea to add the implementation to the main Pulsar
> Admin API at all.
>
> HTTP consuming would be better to handle in a separate component. PIP-64
> doesn't determine that this should be part of Pulsar Admin API and we
> should revisit this decision. I think it's a bad idea to add HTTP consuming
> to Pulsar Admin API and brokers.
>
> I want to discuss whether we should implement the HTTP endpoint in the
> broker or separate it at another component(like pulsar-WebSocket).
>
> Best,
>
> Mattison
>