You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "junwen yang (Jira)" <ji...@apache.org> on 2020/12/02 05:21:00 UTC
[jira] [Updated] (YARN-10508) Protobuf Mesage Incompatibility
Detector
[ https://issues.apache.org/jira/browse/YARN-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
junwen yang updated YARN-10508:
-------------------------------
Description:
Regarding the issue YARN-5632 caused by the incompatibility of protobuf message, we have created a static checker which keeps track of the proto file change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also, the [protobuf guidelines |https://developers.google.com/protocol-buffers/docs/proto]suggests _each field in the message definition has a *unique number*. These numbers are used to identify your fields in the [message binary format|https://developers.google.com/protocol-buffers/docs/encoding], and should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines in [protobuf official website|https://developers.google.com/protocol-buffers/docs/proto], _*Required Is Forever* You should be very careful about marking fields as {{required}}. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field - old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead._
We have applied our checker on the frequently maintained YARN versions: rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0, rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0, rel/release-3.3.0, and we found 28 potential problems as attached.
The results reported by our checker got confirmed by developers of HBASE and our checker is requested by them, which can be found at HBASE-25340.
was:
Regarding the issue YARN-5632 caused by the incompatibility of protobuf message, we have created a static checker which keeps track of the proto file change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also, the [protobuf guidelines |https://developers.google.com/protocol-buffers/docs/proto]suggests _each field in the message definition has a *unique number*. These numbers are used to identify your fields in the [message binary format|https://developers.google.com/protocol-buffers/docs/encoding], and should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines in [protobuf official website|https://developers.google.com/protocol-buffers/docs/proto], _*Required Is Forever* You should be very careful about marking fields as {{required}}. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field - old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead._
We have applied our checker on the frequently maintained YARN versions: rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0, rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0, rel/release-3.3.0, and we found 28 potential problems as attached.
The results reported by our checker got confirmation from developers in HBASE, which can be found here HBASE-25340.
> Protobuf Mesage Incompatibility Detector
> ----------------------------------------
>
> Key: YARN-10508
> URL: https://issues.apache.org/jira/browse/YARN-10508
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: junwen yang
> Priority: Major
> Attachments: yarn_proto_incompatibility.txt
>
>
> Regarding the issue YARN-5632 caused by the incompatibility of protobuf message, we have created a static checker which keeps track of the proto file change, and detect potential incompatibility:
> # Add/delete required field, which is the case reported in HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788. Also, the [protobuf guidelines |https://developers.google.com/protocol-buffers/docs/proto]suggests _each field in the message definition has a *unique number*. These numbers are used to identify your fields in the [message binary format|https://developers.google.com/protocol-buffers/docs/encoding], and should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines in [protobuf official website|https://developers.google.com/protocol-buffers/docs/proto], _*Required Is Forever* You should be very careful about marking fields as {{required}}. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field - old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained YARN versions: rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0, rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0, rel/release-3.3.0, and we found 28 potential problems as attached.
> The results reported by our checker got confirmed by developers of HBASE and our checker is requested by them, which can be found at HBASE-25340.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org