You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Sijie Guo <gu...@gmail.com> on 2020/05/22 17:52:30 UTC
[DISCUSS] Protobuf 2.4.1
Hi all,
I would like to kick off a discussion about what we should do for protobuf
2.4.1 and figure out what is the long term plan.
Currently, Pulsar is using a customized version of protobuf 2.4.1 for wire
protocol. The customization was made to leverage netty object pooling to
reduce object generation and cause bad JVM GC behavior. However, protobuf
2.4.1 is a many-years old release and there is also a vulnerability issue
reported for protobuf-java 2.4.1 (
https://nvd.nist.gov/vuln/detail/CVE-2015-5237). We need to figure out a
plan to upgrade this dependency.
Here are some thoughts. Happy to see what others think.
a) We upgrade to protobuf 3.x without customization.
- Pros: be able to catch up with changes in protobuf and avoid any
vulnerabilities.
- Cons: we might have to suffer the pains from object generations. But it
might not be too bad since we don't store the message payloads in protobuf.
b) We upgrade to protobuf 3.x with new customization.
- Pros: We are able to control object generations.
- Cons: Maintaining a customization makes harder to catch up changes in
protobuf community and avoid vulnerabilities.
c) Look for other alternatives to protobuf such as flatbuffers.
- Cons: it is not a backward-compatible solution with existing Pulsar
clients.
- Sijie
Re: [DISCUSS] Protobuf 2.4.1
Posted by Dave Fisher <wa...@apache.org>.
> On May 22, 2020, at 10:52 AM, Sijie Guo <gu...@gmail.com> wrote:
>
> Hi all,
>
> I would like to kick off a discussion about what we should do for protobuf
> 2.4.1 and figure out what is the long term plan.
>
> Currently, Pulsar is using a customized version of protobuf 2.4.1 for wire
> protocol. The customization was made to leverage netty object pooling to
> reduce object generation and cause bad JVM GC behavior. However, protobuf
> 2.4.1 is a many-years old release and there is also a vulnerability issue
> reported for protobuf-java 2.4.1 (
> https://nvd.nist.gov/vuln/detail/CVE-2015-5237). We need to figure out a
> plan to upgrade this dependency.
>
> Here are some thoughts. Happy to see what others think.
>
> a) We upgrade to protobuf 3.x without customization.
>
> - Pros: be able to catch up with changes in protobuf and avoid any
> vulnerabilities.
> - Cons: we might have to suffer the pains from object generations. But it
> might not be too bad since we don't store the message payloads in protobuf.
>
> b) We upgrade to protobuf 3.x with new customization.
>
> - Pros: We are able to control object generations.
> - Cons: Maintaining a customization makes harder to catch up changes in
> protobuf community and avoid vulnerabilities.
Is it possible that the protobuf community would want us to upstream the customizations?
Regards,
Dave
>
> c) Look for other alternatives to protobuf such as flatbuffers.
>
> - Cons: it is not a backward-compatible solution with existing Pulsar
> clients.
>
> - Sijie
Re: [DISCUSS] Protobuf 2.4.1
Posted by Sijie Guo <gu...@gmail.com>.
Protobuf3 support protobuf2 syntax.
- Sijie
On Fri, May 22, 2020 at 12:27 PM Ming Luo <mi...@kafkaesque.io.invalid>
wrote:
> Hi Sijie,
>
> I just want to clarify if it is to upgrade protobuf 2 to 3. Would that lead
> to a backward compatibility issue for clients? Proto3 has removed the
> custom default, which pulsar api proto uses, and a few others in proto 2.
> Would that be a concern for clients?
>
> Ming
>
> On Fri, 22 May 2020 at 13:52, Sijie Guo <gu...@gmail.com> wrote:
>
> > Hi all,
> >
> > I would like to kick off a discussion about what we should do for
> protobuf
> > 2.4.1 and figure out what is the long term plan.
> >
> > Currently, Pulsar is using a customized version of protobuf 2.4.1 for
> wire
> > protocol. The customization was made to leverage netty object pooling to
> > reduce object generation and cause bad JVM GC behavior. However, protobuf
> > 2.4.1 is a many-years old release and there is also a vulnerability issue
> > reported for protobuf-java 2.4.1 (
> > https://nvd.nist.gov/vuln/detail/CVE-2015-5237). We need to figure out a
> > plan to upgrade this dependency.
> >
> > Here are some thoughts. Happy to see what others think.
> >
> > a) We upgrade to protobuf 3.x without customization.
> >
> > - Pros: be able to catch up with changes in protobuf and avoid any
> > vulnerabilities.
> > - Cons: we might have to suffer the pains from object generations. But it
> > might not be too bad since we don't store the message payloads in
> protobuf.
> >
> > b) We upgrade to protobuf 3.x with new customization.
> >
> > - Pros: We are able to control object generations.
> > - Cons: Maintaining a customization makes harder to catch up changes in
> > protobuf community and avoid vulnerabilities.
> >
> > c) Look for other alternatives to protobuf such as flatbuffers.
> >
> > - Cons: it is not a backward-compatible solution with existing Pulsar
> > clients.
> >
> > - Sijie
> >
>
>
> --
> Powered by Pulsar
> Engineering, Kafkaesque
>
Re: [DISCUSS] Protobuf 2.4.1
Posted by Ming Luo <mi...@kafkaesque.io.INVALID>.
Hi Sijie,
I just want to clarify if it is to upgrade protobuf 2 to 3. Would that lead
to a backward compatibility issue for clients? Proto3 has removed the
custom default, which pulsar api proto uses, and a few others in proto 2.
Would that be a concern for clients?
Ming
On Fri, 22 May 2020 at 13:52, Sijie Guo <gu...@gmail.com> wrote:
> Hi all,
>
> I would like to kick off a discussion about what we should do for protobuf
> 2.4.1 and figure out what is the long term plan.
>
> Currently, Pulsar is using a customized version of protobuf 2.4.1 for wire
> protocol. The customization was made to leverage netty object pooling to
> reduce object generation and cause bad JVM GC behavior. However, protobuf
> 2.4.1 is a many-years old release and there is also a vulnerability issue
> reported for protobuf-java 2.4.1 (
> https://nvd.nist.gov/vuln/detail/CVE-2015-5237). We need to figure out a
> plan to upgrade this dependency.
>
> Here are some thoughts. Happy to see what others think.
>
> a) We upgrade to protobuf 3.x without customization.
>
> - Pros: be able to catch up with changes in protobuf and avoid any
> vulnerabilities.
> - Cons: we might have to suffer the pains from object generations. But it
> might not be too bad since we don't store the message payloads in protobuf.
>
> b) We upgrade to protobuf 3.x with new customization.
>
> - Pros: We are able to control object generations.
> - Cons: Maintaining a customization makes harder to catch up changes in
> protobuf community and avoid vulnerabilities.
>
> c) Look for other alternatives to protobuf such as flatbuffers.
>
> - Cons: it is not a backward-compatible solution with existing Pulsar
> clients.
>
> - Sijie
>
--
Powered by Pulsar
Engineering, Kafkaesque