You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Neng Lu <nl...@apache.org> on 2023/03/01 02:18:06 UTC
Re: [DISCUSS] Support custom compressionType for pulsar functions
+1 for the change.
On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> Hello, community:
>
> ### Motivation
>
> Currently, pulsar functions are using `LZ4` as the compression type, and
> users cannot change it, yet some users may want to custom this behavior.
>
> ### Modifications
>
> Add a `CompressionType` field(which is an enum) to the `ProducerSpec` in
> the `Function.proto`; this enum has six values: `NOTSET`, `NONE`, `LZ4`,
> `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> supported compression type, so that even users don't set the
> `CompressionType`, it will fallback to its "zero" value: `NOTSET` instead
> of `NONE`, and in such case, pulsar function instances will use `LZ4` to
> keep the same behavior with before.
>
> PTAL when you have time and feel free to leave any comments.
>
> Best Regards,
> Pengcheng Jiang
>
> [0] https://github.com/apache/pulsar/pull/19470
> --
>
> <https://streamnative.io/>
>
> Pengcheng Jiang
>
> Software Engineer
>
> e: pengcheng.jiang@streamnative.io
>
> p: 13540631948
>
> streamnative.io
>
> <http://github.com/streamnative>
> <https://www.linkedin.com/company/streamnative/>
> <https://twitter.com/streamnativeio/>
>
Re: [DISCUSS] Support custom compressionType for pulsar functions
Posted by Pengcheng Jiang <pe...@streamnative.io.INVALID>.
Use `LZ4` as zero value seems better, I will update
Rui Fu <rf...@apache.org> 于2023年3月1日周三 11:12写道：
> +1, very useful, just one question: why not set the `LZ4` as the “zero”
> value instead? Like for the enum with following orders: `LZ4`, `NONE`,
> `ZLIB`, `ZSTD` and `SNAPPY`? So it will remain the backward compatibility.
>
> Best,
>
> Rui Fu
> On Mar 1, 2023 at 10:18 +0800, Neng Lu <nl...@apache.org>, wrote:
> > +1 for the change.
> >
> > On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> > > Hello, community:
> > >
> > > ### Motivation
> > >
> > > Currently, pulsar functions are using `LZ4` as the compression type,
> and
> > > users cannot change it, yet some users may want to custom this
> behavior.
> > >
> > > ### Modifications
> > >
> > > Add a `CompressionType` field(which is an enum) to the `ProducerSpec`
> in
> > > the `Function.proto`; this enum has six values: `NOTSET`, `NONE`,
> `LZ4`,
> > > `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> > > supported compression type, so that even users don't set the
> > > `CompressionType`, it will fallback to its "zero" value: `NOTSET`
> instead
> > > of `NONE`, and in such case, pulsar function instances will use `LZ4`
> to
> > > keep the same behavior with before.
> > >
> > > PTAL when you have time and feel free to leave any comments.
> > >
> > > Best Regards,
> > > Pengcheng Jiang
> > >
> > > [0] https://github.com/apache/pulsar/pull/19470
> > > --
> > >
> > > <https://streamnative.io/>
> > >
> > > Pengcheng Jiang
> > >
> > > Software Engineer
> > >
> > > e: pengcheng.jiang@streamnative.io
> > >
> > > p: 13540631948
> > >
> > > streamnative.io
> > >
> > > <http://github.com/streamnative>
> > > <https://www.linkedin.com/company/streamnative/>
> > > <https://twitter.com/streamnativeio/>
> > >
>
--
<https://streamnative.io/>
Pengcheng Jiang
Software Engineer
e: pengcheng.jiang@streamnative.io
p: 13540631948
streamnative.io
<http://github.com/streamnative>
<https://www.linkedin.com/company/streamnative/>
<https://twitter.com/streamnativeio/>
Re: [DISCUSS] Support custom compressionType for pulsar functions
Posted by Rui Fu <rf...@apache.org>.
+1, very useful, just one question: why not set the `LZ4` as the “zero” value instead? Like for the enum with following orders: `LZ4`, `NONE`, `ZLIB`, `ZSTD` and `SNAPPY`? So it will remain the backward compatibility.
Best,
Rui Fu
On Mar 1, 2023 at 10:18 +0800, Neng Lu <nl...@apache.org>, wrote:
> +1 for the change.
>
> On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> > Hello, community:
> >
> > ### Motivation
> >
> > Currently, pulsar functions are using `LZ4` as the compression type, and
> > users cannot change it, yet some users may want to custom this behavior.
> >
> > ### Modifications
> >
> > Add a `CompressionType` field(which is an enum) to the `ProducerSpec` in
> > the `Function.proto`; this enum has six values: `NOTSET`, `NONE`, `LZ4`,
> > `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> > supported compression type, so that even users don't set the
> > `CompressionType`, it will fallback to its "zero" value: `NOTSET` instead
> > of `NONE`, and in such case, pulsar function instances will use `LZ4` to
> > keep the same behavior with before.
> >
> > PTAL when you have time and feel free to leave any comments.
> >
> > Best Regards,
> > Pengcheng Jiang
> >
> > [0] https://github.com/apache/pulsar/pull/19470
> > --
> >
> > <https://streamnative.io/>
> >
> > Pengcheng Jiang
> >
> > Software Engineer
> >
> > e: pengcheng.jiang@streamnative.io
> >
> > p: 13540631948
> >
> > streamnative.io
> >
> > <http://github.com/streamnative>
> > <https://www.linkedin.com/company/streamnative/>
> > <https://twitter.com/streamnativeio/>
> >