You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ashish Khatkar <ak...@yelp.com.INVALID> on 2023/08/04 11:51:12 UTC
Plans for Schema Evolution in Table API
Hi all,
We are using flink-1.17.0 table API and RocksDB as backend to provide a
service to our users to run sql queries. The tables are created using the
avro schema and when the schema is changed in a compatible manner i.e
adding a field with default, we are unable to recover the job from the
savepoint. This is mentioned in the flink doc on evolution [1] as well.
Are there any plans to support schema evolution in the table API? Our
current approach involves rebuilding the entire state by discarding the
output and then utilizing that state in the actual job. This is already
done for table-store [2]
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
Re: Plans for Schema Evolution in Table API
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Ashish,
Table API has many restrictions. Even if the source schema has been changed
in a compatible manner from the database's perspective, the added column
should not be used in the downstream operators, e.g. join. In case schema
evolution is required in your business, you might want to consider using
DataStream API.
Best regards,
Jing
On Wed, Aug 16, 2023 at 12:10 AM liu ron <ro...@gmail.com> wrote:
> Hi, Ashish
>
> As Timo said, the community currently doesn't have plan to support schema
> evolution in Table API,
>
> Best,
> Ron
>
> Timo Walther <tw...@apache.org> 于2023年8月15日周二 23:29写道:
>
> > Hi Ashish,
> >
> > sorry for the late reply. There are currently no concrete plans to
> > support schema evolution in Table API. Until recently, Flink version
> > evolution was the biggest topic. In the near future we can rediscuss
> > query and state evolution in more detail.
> >
> > Personally, I think we will need either some kind of more flexible data
> > type (similar like the JSON type in Postgres) or user-defined types
> > (UDT) to ensure a smooth experience.
> >
> > For now, warming up the state is the only viable solution until internal
> > serializers are more flexible.
> >
> > Regards,
> > Timo
> >
> > On 14.08.23 16:55, Ashish Khatkar wrote:
> > > Bumping the thread.
> > >
> > > On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com>
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> We are using flink-1.17.0 table API and RocksDB as backend to provide
> a
> > >> service to our users to run sql queries. The tables are created using
> > the
> > >> avro schema and when the schema is changed in a compatible manner i.e
> > >> adding a field with default, we are unable to recover the job from the
> > >> savepoint. This is mentioned in the flink doc on evolution [1] as
> well.
> > >>
> > >> Are there any plans to support schema evolution in the table API? Our
> > >> current approach involves rebuilding the entire state by discarding
> the
> > >> output and then utilizing that state in the actual job. This is
> already
> > >> done for table-store [2]
> > >>
> > >> [1]
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> > >> [2]
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
> > >>
> > >>
> > >>
> > >
> >
> >
>
Re: Plans for Schema Evolution in Table API
Posted by liu ron <ro...@gmail.com>.
Hi, Ashish
As Timo said, the community currently doesn't have plan to support schema
evolution in Table API,
Best,
Ron
Timo Walther <tw...@apache.org> 于2023年8月15日周二 23:29写道:
> Hi Ashish,
>
> sorry for the late reply. There are currently no concrete plans to
> support schema evolution in Table API. Until recently, Flink version
> evolution was the biggest topic. In the near future we can rediscuss
> query and state evolution in more detail.
>
> Personally, I think we will need either some kind of more flexible data
> type (similar like the JSON type in Postgres) or user-defined types
> (UDT) to ensure a smooth experience.
>
> For now, warming up the state is the only viable solution until internal
> serializers are more flexible.
>
> Regards,
> Timo
>
> On 14.08.23 16:55, Ashish Khatkar wrote:
> > Bumping the thread.
> >
> > On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com>
> wrote:
> >
> >> Hi all,
> >>
> >> We are using flink-1.17.0 table API and RocksDB as backend to provide a
> >> service to our users to run sql queries. The tables are created using
> the
> >> avro schema and when the schema is changed in a compatible manner i.e
> >> adding a field with default, we are unable to recover the job from the
> >> savepoint. This is mentioned in the flink doc on evolution [1] as well.
> >>
> >> Are there any plans to support schema evolution in the table API? Our
> >> current approach involves rebuilding the entire state by discarding the
> >> output and then utilizing that state in the actual job. This is already
> >> done for table-store [2]
> >>
> >> [1]
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> >> [2]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
> >>
> >>
> >>
> >
>
>
Re: Plans for Schema Evolution in Table API
Posted by Timo Walther <tw...@apache.org>.
Hi Ashish,
sorry for the late reply. There are currently no concrete plans to
support schema evolution in Table API. Until recently, Flink version
evolution was the biggest topic. In the near future we can rediscuss
query and state evolution in more detail.
Personally, I think we will need either some kind of more flexible data
type (similar like the JSON type in Postgres) or user-defined types
(UDT) to ensure a smooth experience.
For now, warming up the state is the only viable solution until internal
serializers are more flexible.
Regards,
Timo
On 14.08.23 16:55, Ashish Khatkar wrote:
> Bumping the thread.
>
> On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com> wrote:
>
>> Hi all,
>>
>> We are using flink-1.17.0 table API and RocksDB as backend to provide a
>> service to our users to run sql queries. The tables are created using the
>> avro schema and when the schema is changed in a compatible manner i.e
>> adding a field with default, we are unable to recover the job from the
>> savepoint. This is mentioned in the flink doc on evolution [1] as well.
>>
>> Are there any plans to support schema evolution in the table API? Our
>> current approach involves rebuilding the entire state by discarding the
>> output and then utilizing that state in the actual job. This is already
>> done for table-store [2]
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
>> [2]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
>>
>>
>>
>
Re: Plans for Schema Evolution in Table API
Posted by Ashish Khatkar <ak...@yelp.com.INVALID>.
Bumping the thread.
On Fri, Aug 4, 2023 at 12:51 PM Ashish Khatkar <ak...@yelp.com> wrote:
> Hi all,
>
> We are using flink-1.17.0 table API and RocksDB as backend to provide a
> service to our users to run sql queries. The tables are created using the
> avro schema and when the schema is changed in a compatible manner i.e
> adding a field with default, we are unable to recover the job from the
> savepoint. This is mentioned in the flink doc on evolution [1] as well.
>
> Are there any plans to support schema evolution in the table API? Our
> current approach involves rebuilding the entire state by discarding the
> output and then utilizing that state in the actual job. This is already
> done for table-store [2]
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store
>
>
>