You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ashish Khatkar <ak...@yelp.com.INVALID> on 2023/08/04 11:51:12 UTC

Plans for Schema Evolution in Table API

Hi all,

We are using flink-1.17.0 table API and RocksDB as backend to provide a
service to our users to run sql queries. The tables are created using the
avro schema and when the schema is changed in a compatible manner i.e
adding a field with default, we are unable to recover the job from the
savepoint. This is mentioned in the flink doc on evolution [1] as well.

Are there any plans to support schema evolution in the table API? Our
current approach involves rebuilding the entire state by discarding the
output and then utilizing that state in the actual job. This is already
done for table-store [2]

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store

Re: Plans for Schema Evolution in Table API

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Ashish,

Table API has many restrictions. Even if the source schema has been changed
in a compatible manner from the database's perspective, the added column
should not be used in the downstream operators, e.g. join. In case schema
evolution is required in your business, you might want to consider using
DataStream API.

Best regards,
Jing

On Wed, Aug 16, 2023 at 12:10 AM liu ron <ro...@gmail.com> wrote:

> Hi, Ashish
>
> As Timo said, the community currently doesn't have plan to support schema
> evolution in Table API,
>
> Best,
> Ron
>
> Timo Walther <tw...@apache.org> 于2023年8月15日周二 23:29写道:
>
> > Hi Ashish,
> >
> > sorry for the late reply. There are currently no concrete plans to
> > support schema evolution in Table API. Until recently, Flink version
> > evolution was the biggest topic. In the near future we can rediscuss
> > query and state evolution in more detail.
> >
> > Personally, I think we will need either some kind of more flexible data
> > type (similar like the JSON type in Postgres) or user-defined types
> > (UDT) to ensure a smooth experience.
> >
> > For now, warming up the state is the only viable solution until internal
> > serializers are more flexible.
> >
> > Regards,
> > Timo
> >
> > On 14.08.23 16:55, Ashish Khatkar wrote:
> > > Bumping the thread.
> > >
> > > On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com>
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> We are using flink-1.17.0 table API and RocksDB as backend to provide
> a
> > >> service to our users to run sql queries. The tables are created using
> > the
> > >> avro schema and when the schema is changed in a compatible manner i.e
> > >> adding a field with default, we are unable to recover the job from the
> > >> savepoint. This is mentioned in the flink doc on evolution [1] as
> well.
> > >>
> > >> Are there any plans to support schema evolution in the table API? Our
> > >> current approach involves rebuilding the entire state by discarding
> the
> > >> output and then utilizing that state in the actual job. This is
> already
> > >> done for table-store [2]
> > >>
> > >> [1]
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> > >> [2]
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
> > >>
> > >>
> > >>
> > >
> >
> >
>

Re: Plans for Schema Evolution in Table API

Posted by liu ron <ro...@gmail.com>.
Hi, Ashish

As Timo said, the community currently doesn't have plan to support schema
evolution in Table API,

Best,
Ron

Timo Walther <tw...@apache.org> 于2023年8月15日周二 23:29写道:

> Hi Ashish,
>
> sorry for the late reply. There are currently no concrete plans to
> support schema evolution in Table API. Until recently, Flink version
> evolution was the biggest topic. In the near future we can rediscuss
> query and state evolution in more detail.
>
> Personally, I think we will need either some kind of more flexible data
> type (similar like the JSON type in Postgres) or user-defined types
> (UDT) to ensure a smooth experience.
>
> For now, warming up the state is the only viable solution until internal
> serializers are more flexible.
>
> Regards,
> Timo
>
> On 14.08.23 16:55, Ashish Khatkar wrote:
> > Bumping the thread.
> >
> > On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com>
> wrote:
> >
> >> Hi all,
> >>
> >> We are using flink-1.17.0 table API and RocksDB as backend to provide a
> >> service to our users to run sql queries. The tables are created using
> the
> >> avro schema and when the schema is changed in a compatible manner i.e
> >> adding a field with default, we are unable to recover the job from the
> >> savepoint. This is mentioned in the flink doc on evolution [1] as well.
> >>
> >> Are there any plans to support schema evolution in the table API? Our
> >> current approach involves rebuilding the entire state by discarding the
> >> output and then utilizing that state in the actual job. This is already
> >> done for table-store [2]
> >>
> >> [1]
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> >> [2]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
> >>
> >>
> >>
> >
>
>

Re: Plans for Schema Evolution in Table API

Posted by Timo Walther <tw...@apache.org>.
Hi Ashish,

sorry for the late reply. There are currently no concrete plans to 
support schema evolution in Table API. Until recently, Flink version 
evolution was the biggest topic. In the near future we can rediscuss 
query and state evolution in more detail.

Personally, I think we will need either some kind of more flexible data 
type (similar like the JSON type in Postgres) or user-defined types 
(UDT) to ensure a smooth experience.

For now, warming up the state is the only viable solution until internal 
serializers are more flexible.

Regards,
Timo

On 14.08.23 16:55, Ashish Khatkar wrote:
> Bumping the thread.
> 
> On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com> wrote:
> 
>> Hi all,
>>
>> We are using flink-1.17.0 table API and RocksDB as backend to provide a
>> service to our users to run sql queries. The tables are created using the
>> avro schema and when the schema is changed in a compatible manner i.e
>> adding a field with default, we are unable to recover the job from the
>> savepoint. This is mentioned in the flink doc on evolution [1] as well.
>>
>> Are there any plans to support schema evolution in the table API? Our
>> current approach involves rebuilding the entire state by discarding the
>> output and then utilizing that state in the actual job. This is already
>> done for table-store [2]
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
>> [2]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
>>
>>
>>
> 


Re: Plans for Schema Evolution in Table API

Posted by Ashish Khatkar <ak...@yelp.com.INVALID>.
Bumping the thread.

On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com> wrote:

> Hi all,
>
> We are using flink-1.17.0 table API and RocksDB as backend to provide a
> service to our users to run sql queries. The tables are created using the
> avro schema and when the schema is changed in a compatible manner i.e
> adding a field with default, we are unable to recover the job from the
> savepoint. This is mentioned in the flink doc on evolution [1] as well.
>
> Are there any plans to support schema evolution in the table API? Our
> current approach involves rebuilding the entire state by discarding the
> output and then utilizing that state in the actual job. This is already
> done for table-store [2]
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
>
>
>