You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Hangxiang Yu <ma...@gmail.com> on 2022/10/12 12:41:07 UTC

[DISCUSS] FLIP-263: Improve resolving schema compatibility

Dear Flink developers,

I would like to start a discussion thread on FLIP-263[1] proposing to
improve the usability of resolving schema compatibility.

Currently, the place for compatibility checks is
TypeSerializerSnapshot#resolveSchemaCompatibility
which belongs to the old serializer, There are no ways for users to specify the
compatibility with the old serializer in the new customized serializer.

The FLIP hopes to reverse the direction of resolving schema compatibility
to improve the usability of resolving schema compatibility.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Posted by Hangxiang Yu <ma...@gmail.com>.

Hi, Zakelly.
Thanks for your suggestion and I think it makes sense.

I think both two options should be adopted. The option 2 is just what I
mentioned in step 2. At the same time, we still need option 1 because we
should make sure the existing user code could run normally without
modifying any codes before we remove the deprecated method (
TypeSerializerSnapshot#resolveSchemaCompatibility).


Hi, Yanfei.
Thanks for your feedback.

Will Typeserializer#resolveSchemaCompatibility become an abstract method
that must be implemented here?

Considering the compatibility before removing the deprecated method (
TypeSerializerSnapshot#resolveSchemaCompatibility), we may still need to
have a default implementation for
Typeserializer#resolveSchemaCompatibility. When we remove the deprecated
method (In the step 3), I think it deserves a discussion whether we need to
make it abstract or not (it will break changes twice).

Does it support the migration from built-in serializer -> custom serializer
-> built-in serializer ?

I think after reversing the direction of resolving schema compatibility,
users could just consider the compatibility result in the newest
serializer. If the built-in serializer cannot work well, users could just
specify their new serializer. It could make users always have ways to
restore from the snapshot if they want to do so.

On Thu, Oct 13, 2022 at 11:37 AM yanfei lei <fr...@gmail.com> wrote:

> Hi Hangxiang,
>
>
> Thanks for raising the discussion, +1 for reversing the direction of
> resolving schema compatibility.
>
> As you described, in 'Step 1', Typeserializer#resolveSchemaCompatibility
> will return TYPE.INCOMPATIBLE default,
> Typeserializer#resolveSchemaCompatibility is a default method; in 'Step
> 2&3', you proposed deprecating
> TypeSerializerSnapshot#resolveSchemaCompatibility for the long run, will
> Typeserializer#resolveSchemaCompatibility become an abstract method that
> must be implemented here?
>
>
> After this FILP， the new serializer claims its compatibility with the old
> serializer， does it support the migration from
>
> built-in serializer -> custom serializer -> built-in serializer ?
>
> as you mentioned, some built-in serializers are final class, we can't
> change the #resolveSchemaCompatibility() method of them.
>
>
> Best,
>
> Yanfei
>
> Zakelly Lan <za...@gmail.com> 于2022年10月13日周四 00:56写道：
>
> > Hi Hangxiang,
> >
> > Thanks for driving this. It is reasonable to let the new serializer
> > claim its compatibility with the old serializer. However, I think
> > there is a little confusion as you described in your proposed change
> > 'Step 1'. You mean that we let the new serializer check the
> > compatibility first, and if it gives the 'INCOMPATIBLE' then we let
> > the old serializer do the check as before. The 'INCOMPATIBLE' from the
> > new serializer means 'do not know' or 'unresolved' here, but sometimes
> > the new serializer should express a decisive meaning that it is really
> > not compatible with the older one, which is the 'INCOMPATIBLE' should
> > mean. There is a semantic ambiguity or missing. I think there are two
> > options to resolve this:
> > 1. Provide 'UNRESOLVED' in type of the compatibility checking result,
> > and let the proposed new interface return it by default. OR
> > 2. Replace any use of the old interface with the new interface. In the
> > default implementation of the new interface, call the old interface to
> > leverage the old result. This approach provides the ability to totally
> > replace original checking logic (by implementing the new interface in
> > new serializer) while maintaining good backward compatibility.
> >
> > What do you think?
> >
> > Best,
> > Zakelly.
> >
> >
> > On Wed, Oct 12, 2022 at 8:41 PM Hangxiang Yu <ma...@gmail.com>
> wrote:
> > >
> > > Dear Flink developers,
> > >
> > > I would like to start a discussion thread on FLIP-263[1] proposing to
> > > improve the usability of resolving schema compatibility.
> > >
> > > Currently, the place for compatibility checks is
> > > TypeSerializerSnapshot#resolveSchemaCompatibility
> > > which belongs to the old serializer, There are no ways for users to
> > specify the
> > > compatibility with the old serializer in the new customized serializer.
> > >
> > > The FLIP hopes to reverse the direction of resolving schema
> compatibility
> > > to improve the usability of resolving schema compatibility.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Posted by yanfei lei <fr...@gmail.com>.

Hi Hangxiang,


Thanks for raising the discussion, +1 for reversing the direction of
resolving schema compatibility.

As you described, in 'Step 1', Typeserializer#resolveSchemaCompatibility
will return TYPE.INCOMPATIBLE default,
Typeserializer#resolveSchemaCompatibility is a default method; in 'Step
2&3', you proposed deprecating
TypeSerializerSnapshot#resolveSchemaCompatibility for the long run, will
Typeserializer#resolveSchemaCompatibility become an abstract method that
must be implemented here?


After this FILP， the new serializer claims its compatibility with the old
serializer， does it support the migration from

built-in serializer -> custom serializer -> built-in serializer ?

as you mentioned, some built-in serializers are final class, we can't
change the #resolveSchemaCompatibility() method of them.


Best,

Yanfei

Zakelly Lan <za...@gmail.com> 于2022年10月13日周四 00:56写道：

> Hi Hangxiang,
>
> Thanks for driving this. It is reasonable to let the new serializer
> claim its compatibility with the old serializer. However, I think
> there is a little confusion as you described in your proposed change
> 'Step 1'. You mean that we let the new serializer check the
> compatibility first, and if it gives the 'INCOMPATIBLE' then we let
> the old serializer do the check as before. The 'INCOMPATIBLE' from the
> new serializer means 'do not know' or 'unresolved' here, but sometimes
> the new serializer should express a decisive meaning that it is really
> not compatible with the older one, which is the 'INCOMPATIBLE' should
> mean. There is a semantic ambiguity or missing. I think there are two
> options to resolve this:
> 1. Provide 'UNRESOLVED' in type of the compatibility checking result,
> and let the proposed new interface return it by default. OR
> 2. Replace any use of the old interface with the new interface. In the
> default implementation of the new interface, call the old interface to
> leverage the old result. This approach provides the ability to totally
> replace original checking logic (by implementing the new interface in
> new serializer) while maintaining good backward compatibility.
>
> What do you think?
>
> Best,
> Zakelly.
>
>
> On Wed, Oct 12, 2022 at 8:41 PM Hangxiang Yu <ma...@gmail.com> wrote:
> >
> > Dear Flink developers,
> >
> > I would like to start a discussion thread on FLIP-263[1] proposing to
> > improve the usability of resolving schema compatibility.
> >
> > Currently, the place for compatibility checks is
> > TypeSerializerSnapshot#resolveSchemaCompatibility
> > which belongs to the old serializer, There are no ways for users to
> specify the
> > compatibility with the old serializer in the new customized serializer.
> >
> > The FLIP hopes to reverse the direction of resolving schema compatibility
> > to improve the usability of resolving schema compatibility.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
>

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Posted by Zakelly Lan <za...@gmail.com>.

Hi Hangxiang,

Thanks for driving this. It is reasonable to let the new serializer
claim its compatibility with the old serializer. However, I think
there is a little confusion as you described in your proposed change
'Step 1'. You mean that we let the new serializer check the
compatibility first, and if it gives the 'INCOMPATIBLE' then we let
the old serializer do the check as before. The 'INCOMPATIBLE' from the
new serializer means 'do not know' or 'unresolved' here, but sometimes
the new serializer should express a decisive meaning that it is really
not compatible with the older one, which is the 'INCOMPATIBLE' should
mean. There is a semantic ambiguity or missing. I think there are two
options to resolve this:
1. Provide 'UNRESOLVED' in type of the compatibility checking result,
and let the proposed new interface return it by default. OR
2. Replace any use of the old interface with the new interface. In the
default implementation of the new interface, call the old interface to
leverage the old result. This approach provides the ability to totally
replace original checking logic (by implementing the new interface in
new serializer) while maintaining good backward compatibility.

What do you think?

Best,
Zakelly.

On Wed, Oct 12, 2022 at 8:41 PM Hangxiang Yu <ma...@gmail.com> wrote:
>
> Dear Flink developers,
>
> I would like to start a discussion thread on FLIP-263[1] proposing to
> improve the usability of resolving schema compatibility.
>
> Currently, the place for compatibility checks is
> TypeSerializerSnapshot#resolveSchemaCompatibility
> which belongs to the old serializer, There are no ways for users to specify the
> compatibility with the old serializer in the new customized serializer.
>
> The FLIP hopes to reverse the direction of resolving schema compatibility
> to improve the usability of resolving schema compatibility.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility