thrift incompatibility

[junwen yang <x1...@gmail.com>]

Deal all,


Regarding the issue caused by the incompatibility of thrift message such as
IMPALA-8243 <https://issues.apache.org/jira/browse/IMPALA-8243> , we have
created a static checker which keeps track of the thrift file change, and
detects potential incompatibility:

   1. Add/delete required field.  The thrift guidelines suggest *Any new
   fields that you add should be optional*.
   2. The tag number of a field has been changed. Also, the thrift
   guidelines suggest *Don’t change the numeric tags for any existing
   fields*.
   3. A  required field has been changed to optional, or an optional field
   has been changed to required. According to the guidelines, *Required Is
   Forever** You should be very careful about marking fields as required.
   If at some point you wish to stop writing or sending a required field, it
   will be problematic to change the field to an optional field — old readers
   will consider messages without this field to be incomplete and may reject
   or drop them unintentionally. You should consider writing
   application-specific custom validation routines for your buffers instead.
   Some have come to the conclusion that using required does more harm than
   good; they prefer to use only optional. However, this view is not
   universal.*

We have applied our checker on the frequently maintained Accumulo versions:
1.7.0, rel/1.10.0, rel/2.0.0, rel/2.0.0-alpha-1, rel/2.0.0-alpha-2,
rel/2.0.1, main, we found more than 10 problems as attached.

The results reported by our checker got confirmed by developers of HBASE
and our checker is requested by them, which can be found at HBASE-25340
<https://issues.apache.org/jira/browse/HBASE-25340>.


Best,

Junwen

Re: thrift incompatibility

[Christopher <ct...@apache.org>]

Hi Junwen,

It looks like the issues that your checker found were only problems in some
old alpha releases, and we're already fixed by the time we created a final
release.


On Tue, May 4, 2021, 02:15 junwen yang <x1...@gmail.com> wrote:

> Deal all,
>
>
> Regarding the issue caused by the incompatibility of thrift message such
> as IMPALA-8243 <https://issues.apache.org/jira/browse/IMPALA-8243> , we
> have created a static checker which keeps track of the thrift file change,
> and detects potential incompatibility:
>
>    1. Add/delete required field.  The thrift guidelines suggest *Any new
>    fields that you add should be optional*.
>    2. The tag number of a field has been changed. Also, the thrift
>    guidelines suggest *Don’t change the numeric tags for any existing
>    fields*.
>    3. A  required field has been changed to optional, or an optional
>    field has been changed to required. According to the guidelines, *Required
>    Is Forever** You should be very careful about marking fields as
>    required. If at some point you wish to stop writing or sending a required
>    field, it will be problematic to change the field to an optional field —
>    old readers will consider messages without this field to be incomplete and
>    may reject or drop them unintentionally. You should consider writing
>    application-specific custom validation routines for your buffers instead.
>    Some have come to the conclusion that using required does more harm than
>    good; they prefer to use only optional. However, this view is not
>    universal.*
>
> We have applied our checker on the frequently maintained Accumulo
> versions: 1.7.0, rel/1.10.0, rel/2.0.0, rel/2.0.0-alpha-1,
> rel/2.0.0-alpha-2, rel/2.0.1, main, we found more than 10 problems as
> attached.
>
> The results reported by our checker got confirmed by developers of HBASE
> and our checker is requested by them, which can be found at HBASE-25340
> <https://issues.apache.org/jira/browse/HBASE-25340>.
>
>
> Best,
>
> Junwen
>