You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Sijie Guo <gu...@gmail.com> on 2021/02/22 09:18:58 UTC
[Discuss] Separate Function API from Pub/Sub API for python client
Hi,
Currently, the python pulsar client couples function API together with
Pub/Sub API. That means the python client pulls in a lot of unnecessary
dependencies for people who only use pub/sub API.
As discussed in the community meeting last week, I am starting an email
thread for discussing it. I would like to learn what @Sanjeev Kulkarni
<sa...@splunk.com> and @Jerry Peng <je...@splunk.com> think about it.
- Sijie
Re: [Discuss] Separate Function API from Pub/Sub API for python client
Posted by Sijie Guo <gu...@gmail.com>.
Matteo - Thank you for providing the context!
Yes. It is related to gRPC pulled by the state requirements.
- Sijie
On Mon, Feb 22, 2021 at 12:03 PM Matteo Merli <ma...@gmail.com>
wrote:
> Yes, there are a bunch of dependencies that are only needed for state
> store, such as the bk state store client and everything it's pulling
> in, with the biggest offender being GRPC client lib.
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Mon, Feb 22, 2021 at 11:59 AM Sanjeev Kulkarni
> <sa...@splunk.com.invalid> wrote:
> >
> > IIUC, its because of the various state store related dependencies that
> are
> > polluting this?
> >
> > On Mon, Feb 22, 2021 at 10:48 AM Jerry Peng <jerry.boyang.peng@gmail.com
> >
> > wrote:
> >
> > > [ External sender. Exercise caution. ]
> > >
> > > Hi Sijie,
> > >
> > > Are you referring to the fact that the Pulsar python package (wheel
> > > package) includes both pub/sub API and the function API? I If so, why
> are
> > > there a lot of dependencies being pulled in? For Pulsar Python
> Functions,
> > > shouldn't only the following files be pulled into the python package?
> > >
> > >
> > >
> https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python/pulsar/functions
> > >
> > >
> > > It is a minimal set of files. Is the problem you are describing stating
> > > that users wanting to just run Python Functions having to install a
> package
> > > that also includes the pub/sub API and the Python client? And it would
> be
> > > better for users that just want to run Python Functions to not have to
> also
> > > install the python client?
> > >
> > > Best,
> > >
> > > Jerry
> > >
> > >
> > > On Mon, Feb 22, 2021 at 1:19 AM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> Currently, the python pulsar client couples function API together with
> > >> Pub/Sub API. That means the python client pulls in a lot of
> unnecessary
> > >> dependencies for people who only use pub/sub API.
> > >>
> > >> As discussed in the community meeting last week, I am starting an
> email
> > >> thread for discussing it. I would like to learn what @Sanjeev Kulkarni
> > >> <sa...@splunk.com> and @Jerry Peng <je...@splunk.com> think
> about it.
> > >>
> > >> - Sijie
> > >>
> > >
>
Re: [Discuss] Separate Function API from Pub/Sub API for python client
Posted by Matteo Merli <ma...@gmail.com>.
Yes, there are a bunch of dependencies that are only needed for state
store, such as the bk state store client and everything it's pulling
in, with the biggest offender being GRPC client lib.
--
Matteo Merli
<ma...@gmail.com>
On Mon, Feb 22, 2021 at 11:59 AM Sanjeev Kulkarni
<sa...@splunk.com.invalid> wrote:
>
> IIUC, its because of the various state store related dependencies that are
> polluting this?
>
> On Mon, Feb 22, 2021 at 10:48 AM Jerry Peng <je...@gmail.com>
> wrote:
>
> > [ External sender. Exercise caution. ]
> >
> > Hi Sijie,
> >
> > Are you referring to the fact that the Pulsar python package (wheel
> > package) includes both pub/sub API and the function API? I If so, why are
> > there a lot of dependencies being pulled in? For Pulsar Python Functions,
> > shouldn't only the following files be pulled into the python package?
> >
> >
> > https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python/pulsar/functions
> >
> >
> > It is a minimal set of files. Is the problem you are describing stating
> > that users wanting to just run Python Functions having to install a package
> > that also includes the pub/sub API and the Python client? And it would be
> > better for users that just want to run Python Functions to not have to also
> > install the python client?
> >
> > Best,
> >
> > Jerry
> >
> >
> > On Mon, Feb 22, 2021 at 1:19 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Currently, the python pulsar client couples function API together with
> >> Pub/Sub API. That means the python client pulls in a lot of unnecessary
> >> dependencies for people who only use pub/sub API.
> >>
> >> As discussed in the community meeting last week, I am starting an email
> >> thread for discussing it. I would like to learn what @Sanjeev Kulkarni
> >> <sa...@splunk.com> and @Jerry Peng <je...@splunk.com> think about it.
> >>
> >> - Sijie
> >>
> >
Re: [Discuss] Separate Function API from Pub/Sub API for python client
Posted by Sanjeev Kulkarni <sa...@splunk.com.INVALID>.
IIUC, its because of the various state store related dependencies that are
polluting this?
On Mon, Feb 22, 2021 at 10:48 AM Jerry Peng <je...@gmail.com>
wrote:
> [ External sender. Exercise caution. ]
>
> Hi Sijie,
>
> Are you referring to the fact that the Pulsar python package (wheel
> package) includes both pub/sub API and the function API? I If so, why are
> there a lot of dependencies being pulled in? For Pulsar Python Functions,
> shouldn't only the following files be pulled into the python package?
>
>
> https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python/pulsar/functions
>
>
> It is a minimal set of files. Is the problem you are describing stating
> that users wanting to just run Python Functions having to install a package
> that also includes the pub/sub API and the Python client? And it would be
> better for users that just want to run Python Functions to not have to also
> install the python client?
>
> Best,
>
> Jerry
>
>
> On Mon, Feb 22, 2021 at 1:19 AM Sijie Guo <gu...@gmail.com> wrote:
>
>> Hi,
>>
>> Currently, the python pulsar client couples function API together with
>> Pub/Sub API. That means the python client pulls in a lot of unnecessary
>> dependencies for people who only use pub/sub API.
>>
>> As discussed in the community meeting last week, I am starting an email
>> thread for discussing it. I would like to learn what @Sanjeev Kulkarni
>> <sa...@splunk.com> and @Jerry Peng <je...@splunk.com> think about it.
>>
>> - Sijie
>>
>
Re: [Discuss] Separate Function API from Pub/Sub API for python client
Posted by Jerry Peng <je...@gmail.com>.
Hi Sijie,
Are you referring to the fact that the Pulsar python package (wheel
package) includes both pub/sub API and the function API? I If so, why are
there a lot of dependencies being pulled in? For Pulsar Python Functions,
shouldn't only the following files be pulled into the python package?
https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python/pulsar/functions
It is a minimal set of files. Is the problem you are describing stating
that users wanting to just run Python Functions having to install a package
that also includes the pub/sub API and the Python client? And it would be
better for users that just want to run Python Functions to not have to also
install the python client?
Best,
Jerry
On Mon, Feb 22, 2021 at 1:19 AM Sijie Guo <gu...@gmail.com> wrote:
> Hi,
>
> Currently, the python pulsar client couples function API together with
> Pub/Sub API. That means the python client pulls in a lot of unnecessary
> dependencies for people who only use pub/sub API.
>
> As discussed in the community meeting last week, I am starting an email
> thread for discussing it. I would like to learn what @Sanjeev Kulkarni
> <sa...@splunk.com> and @Jerry Peng <je...@splunk.com> think about it.
>
> - Sijie
>