You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Yong Zhang <zh...@gmail.com> on 2019/10/31 09:03:10 UTC

[ DISCUSS ]PIP-50: Package Management

Hi all,

I am starting a proposal about providing a package management to manage
user functions and connectors.

*motivation*

Apache Pulsar provides Functions for users transforming their data as they want.
If there are many users who use the same function in different namespaces, they 
need to upload the same function multiple times. The same problem exists in Sinks 
and Sources as well. 

Also, there is no version management in Functions, Sinks, and Sources. If a 
new version of a Function is developed, users have to update the function 
with a new package. If bugs are found in the new version of the Function, 
users have to rollback the function by updating the function with an old package. 
That means users have to manage their packages in their own services. 

The detail proposal is in
https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing <https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing>

If you have any issues feel free to let me know.

Thanks,
Yong

Re: [ DISCUSS ]PIP-50: Package Management

Posted by Yong Zhang <zh...@gmail.com>.

Hi Sijie,

Thanks for your suggestion!

I will do it as you suggested.

Thanks,
Yong



On Sat, 2 Nov 2019 at 12:24, Sijie Guo <gu...@gmail.com> wrote:

> Hi Yong,
>
> Thank you for putting this together. The overall approach looks good to me.
> Since we already use bookkeeper for package management, this proposal just
> abstracts the package management into a common facility. That seems totally
> reasonable.
>
> I have two suggestions:
>
> 1) make the package storage pluggable. it makes sense to use bookkeeper as
> the package storage for bare mental deployments.
> however it makes more sense to use cloud storage for this purpose when
> running Pulsar in cloud.
>
> 2) regarding changes in function runtime, there are many if-else in
> handling different package protocols (e.g. http, file).
> we should consider improving that into a better abstraction to make it
> easier to extend to different protocol in the future.
>
> Thanks,
> Sijie
>
> On Thu, Oct 31, 2019 at 5:03 PM Yong Zhang <zh...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I am starting a proposal about providing a package management to manage
> > user functions and connectors.
> >
> > *motivation*
> >
> > Apache Pulsar provides Functions for users transforming their data as
> they
> > want.
> > If there are many users who use the same function in different
> namespaces,
> > they
> > need to upload the same function multiple times. The same problem exists
> > in Sinks
> > and Sources as well.
> >
> > Also, there is no version management in Functions, Sinks, and Sources. If
> > a
> > new version of a Function is developed, users have to update the function
> > with a new package. If bugs are found in the new version of the Function,
> > users have to rollback the function by updating the function with an old
> > package.
> > That means users have to manage their packages in their own services.
> >
> > The detail proposal is in
> >
> >
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> > <
> >
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> > >
> >
> > If you have any issues feel free to let me know.
> >
> > Thanks,
> > Yong
>

Re: [ DISCUSS ]PIP-50: Package Management

Posted by Sijie Guo <gu...@gmail.com>.

Hi Yong,

Thank you for putting this together. The overall approach looks good to me.
Since we already use bookkeeper for package management, this proposal just
abstracts the package management into a common facility. That seems totally
reasonable.

I have two suggestions:

1) make the package storage pluggable. it makes sense to use bookkeeper as
the package storage for bare mental deployments.
however it makes more sense to use cloud storage for this purpose when
running Pulsar in cloud.

2) regarding changes in function runtime, there are many if-else in
handling different package protocols (e.g. http, file).
we should consider improving that into a better abstraction to make it
easier to extend to different protocol in the future.

Thanks,
Sijie

On Thu, Oct 31, 2019 at 5:03 PM Yong Zhang <zh...@gmail.com>
wrote:

> Hi all,
>
> I am starting a proposal about providing a package management to manage
> user functions and connectors.
>
> *motivation*
>
> Apache Pulsar provides Functions for users transforming their data as they
> want.
> If there are many users who use the same function in different namespaces,
> they
> need to upload the same function multiple times. The same problem exists
> in Sinks
> and Sources as well.
>
> Also, there is no version management in Functions, Sinks, and Sources. If
> a
> new version of a Function is developed, users have to update the function
> with a new package. If bugs are found in the new version of the Function,
> users have to rollback the function by updating the function with an old
> package.
> That means users have to manage their packages in their own services.
>
> The detail proposal is in
>
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> <
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> >
>
> If you have any issues feel free to let me know.
>
> Thanks,
> Yong

Re: [ DISCUSS ]PIP-50: Package Management

Posted by Guangning E <eg...@gmail.com>.

I think this feature is very useful and I look forward to it.

Thanks,
Guangning

Yong Zhang <zh...@gmail.com> 于2019年10月31日周四 下午5:03写道：

> Hi all,
>
> I am starting a proposal about providing a package management to manage
> user functions and connectors.
>
> *motivation*
>
> Apache Pulsar provides Functions for users transforming their data as they
> want.
> If there are many users who use the same function in different namespaces,
> they
> need to upload the same function multiple times. The same problem exists
> in Sinks
> and Sources as well.
>
> Also, there is no version management in Functions, Sinks, and Sources. If
> a
> new version of a Function is developed, users have to update the function
> with a new package. If bugs are found in the new version of the Function,
> users have to rollback the function by updating the function with an old
> package.
> That means users have to manage their packages in their own services.
>
> The detail proposal is in
>
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> <
> https://docs.google.com/document/d/1FDSNhc8YB1yUMFBUrx8p-1nH1s5Kftp04PkFKWec-2U/edit?usp=sharing
> >
>
> If you have any issues feel free to let me know.
>
> Thanks,
> Yong