You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "junwen yang (Jira)" <ji...@apache.org> on 2021/03/22 15:37:00 UTC
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility
Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306294#comment-17306294 ]
junwen yang commented on IMPALA-10370:
--------------------------------------
Hi, [~tarmstrong], just a quick follow up on this issue, if you feel this tool is valuable to be integrated into IMPALA, we are more than willing to offer any help for free. We are a research team from UChicago, Purdue, and UToronto, and wish to detect and fix bugs that cause upgrade failure.
[|https://issues.apache.org/jira/secure/AddComment!default.jspa?id=13343143]
> Thrift Message Incompatibility Detector
> ---------------------------------------
>
> Key: IMPALA-10370
> URL: https://issues.apache.org/jira/browse/IMPALA-10370
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Reporter: junwen yang
> Priority: Major
> Attachments: impala_thrift_incompatibility.txt, impala_thrift_incompatibility.txt-v2.txt
>
>
> Regarding the issue IMPALA-8243 caused by the incompatibility of thrift message, we have created a static checker which keeps track of the thrift file change, and detect potential incompatibility:
> # Add/delete required field. The thrift guidelines suggests _Any new fields that you add should be optional_.
> # The tag number of a field has been changed. Also, the thrift guidelines suggests _Don’t change the numeric tags for any existing fields_.
> # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines , _*Required Is Forever* You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come to the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal._
> We have applied our checker on the frequently maintained IMPALA versions: refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found more than 1000 problems as attached.
> The results reported by our checker got confirmed by developers of HBASE and our checker is requested by them, which can be found at HBASE-25340.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org