You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gabor Somogyi <ga...@gmail.com> on 2024/12/18 09:43:44 UTC

[DISCUSS] FLIP-496: SQL connector for keyed state data

Hi All,

I'd like to start a discussion of FLIP-496: SQL connector for keyed state
data [1].
Feel free to add your thoughts to make this feature better.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data

BR,
G

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Agreed, making the adjustments in the FLIP...

On Thu, Dec 19, 2024 at 11:33 AM Zakelly Lan <za...@gmail.com> wrote:

> Hi Gabor,
>
> Yes, that's what I meant.
>
>
> Best,
> Zakelly
>
> On Thu, Dec 19, 2024 at 6:22 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
> > Hi Zakelly,
> >
> > Just for my own clear understanding the suggestion is the following?
> >
> > state.type=keyed|non-keyed|operator|metadata
> >
> > Where non-keyed == operator?
> > I'm fine with this.
> >
> > BR,
> > G
> >
> >
> > On Thu, Dec 19, 2024 at 10:31 AM Zakelly Lan <za...@gmail.com>
> > wrote:
> >
> > > Hi Gabor,
> > >
> > > The proposed configs look good to me. I also suggest that 'operator'
> can
> > > serve as an equivalent to 'non-keyed', as non-keyed state is also
> called
> > > operator state [1].
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#operator-state
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Thu, Dec 19, 2024 at 4:46 PM Gabor Somogyi <
> gabor.g.somogyi@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Fair point to have short names.
> > > > How about having the following configs then?
> > > >
> > > > connector=savepoint
> > > > state.type=keyed|non-keyed|metadata
> > > >
> > > > This would introduce the proper keyed/non-keyed option + the config
> > pair
> > > > will
> > > > exactly tell that savepoint metadata is going to be read.
> > > >
> > > > BR,
> > > > G
> > > >
> > > >
> > > > On Thu, Dec 19, 2024 at 4:11 AM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Gabor,
> > > > >
> > > > > Thanks for your answers!
> > > > >
> > > > > I'm good with naming it 'savepoint' with an extra option like
> 'mode'
> > > > > specifying the reading type, or suffixes in names like
> > > > 'savepoint-metadata'
> > > > > for each type of state. For the second option it means the current
> > > > > connector will be named 'savepoint-keyed'? Well I slightly prefer a
> > > short
> > > > > name.
> > > > >
> > > > > For connector='metadata', it seems unclear to me (what is metadata
> of
> > > > Flink
> > > > > :D), so I don't recommend this option.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > >
> > > > > On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <
> > > > gabor.g.somogyi@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thanks for your comments, please find my answers inline.
> > > > > >
> > > > > > BR,
> > > > > > G
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <
> zakelly.lan@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Gabor,
> > > > > > >
> > > > > > > Thanks for the FLIP! It's very convenient for users to use SQL
> to
> > > > read
> > > > > > > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > > > > > > thoughts/questions:
> > > > > > >
> > > > > > >    1. I see this connector is designed to read the
> > > > checkpoint/savepoint
> > > > > > >    data, so I'd suggest naming this connector as 'checkpoint'
> or
> > > > > > > 'savepoint',
> > > > > > >    which is more intuitive. The "connector='state'" makes me
> > think
> > > > this
> > > > > > is
> > > > > > >    reading the runtime state. Also the 'state.path' can be
> > changed
> > > to
> > > > > > >    'checkpoint.path'.
> > > > > > >
> > > > > > Rename is a possibility, I'm fine with either way in terms of
> > naming.
> > > > > > The only thing what I would consider is that later on it would be
> > > good
> > > > > > to add metadata reading possibility. Such case we can either
> > provide
> > > > > > "savepoint" for data reading and "savepoint_metadata" or simply
> > > > > "metadata"
> > > > > > for meta reading (See 3rd bullet for another approach). Please
> > share
> > > > your
> > > > > > thoughts on this.
> > > > > >
> > > > > >
> > > > > > >    2. IIUC this connector is a bounded source and batch
> execution
> > > is
> > > > > > >    preferred, right? And it cannot read checkpoints while the
> > > > original
> > > > > > job
> > > > > > > is
> > > > > > >    running and doing periodic checkpoints. Thus I'd prefer the
> > name
> > > > > > >    'savepoint', as the lifecycle of savepoints is managed by
> the
> > > user
> > > > > > > instead
> > > > > > >    of the Flink job.
> > > > > > >
> > > > > > We can call it savepoint, just agree on the metadata naming as
> well
> > > to
> > > > > have
> > > > > > an overall picture.
> > > > > >
> > > > > >
> > > > > > >    3. As you pictured, there might be more connectors for
> states
> > > > other
> > > > > > than
> > > > > > >    keyed state, so I'm thinking about providing them in one
> > > connector
> > > > > > with
> > > > > > > an
> > > > > > >    option 'state.type' enumerating 'keyed',  'non-keyed'
> > > > ('operator'),
> > > > > > and
> > > > > > > so
> > > > > > >    on. Well for the first version this is not a required
> option.
> > > > Seems
> > > > > > more
> > > > > > >    organized. WDYT?
> > > > > > >
> > > > > > Yeah, later on we must provide such config parameter somehow.
> > > > > > Couple of things to consider:
> > > > > > - Maybe we can add "metadata" as possible value here instead of
> > > > > > "connector=X"
> > > > > > - I'm thinking about whether we can come up with something
> > > > > > where we can mix metadata, keyed and non-keyed data in a single
> SQL
> > > > > > statement.
> > > > > > Without super deep consideration I would say no, but I'm going to
> > > think
> > > > > it
> > > > > > through.
> > > > > >
> > > > > >
> > > > > > >    4. I'm not sure if the general option 'key.format' is
> needed,
> > > > would
> > > > > > >    using 'fields.k.format' while the 'k' is the name of the SQL
> > > > primary
> > > > > > > column
> > > > > > >    be feasible?
> > > > > > >
> > > > > > I think that's a nice catch of a simplification.
> > > > > >
> > > > > >
> > > > > > >    5. Reading window state[1] is unsupported, right?
> > > > > > >
> > > > > > Yeah, not yet covered here and can come in a later FLIP.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <
> > > > > gabor.g.somogyi@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for
> > > keyed
> > > > > > state
> > > > > > > > data [1].
> > > > > > > > Feel free to add your thoughts to make this feature better.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > > > > > >
> > > > > > > > BR,
> > > > > > > > G
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Zakelly Lan <za...@gmail.com>.
Hi Gabor,

Yes, that's what I meant.


Best,
Zakelly

On Thu, Dec 19, 2024 at 6:22 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi Zakelly,
>
> Just for my own clear understanding the suggestion is the following?
>
> state.type=keyed|non-keyed|operator|metadata
>
> Where non-keyed == operator?
> I'm fine with this.
>
> BR,
> G
>
>
> On Thu, Dec 19, 2024 at 10:31 AM Zakelly Lan <za...@gmail.com>
> wrote:
>
> > Hi Gabor,
> >
> > The proposed configs look good to me. I also suggest that 'operator' can
> > serve as an equivalent to 'non-keyed', as non-keyed state is also called
> > operator state [1].
> >
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#operator-state
> >
> > Best,
> > Zakelly
> >
> > On Thu, Dec 19, 2024 at 4:46 PM Gabor Somogyi <gabor.g.somogyi@gmail.com
> >
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > Fair point to have short names.
> > > How about having the following configs then?
> > >
> > > connector=savepoint
> > > state.type=keyed|non-keyed|metadata
> > >
> > > This would introduce the proper keyed/non-keyed option + the config
> pair
> > > will
> > > exactly tell that savepoint metadata is going to be read.
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Thu, Dec 19, 2024 at 4:11 AM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi Gabor,
> > > >
> > > > Thanks for your answers!
> > > >
> > > > I'm good with naming it 'savepoint' with an extra option like 'mode'
> > > > specifying the reading type, or suffixes in names like
> > > 'savepoint-metadata'
> > > > for each type of state. For the second option it means the current
> > > > connector will be named 'savepoint-keyed'? Well I slightly prefer a
> > short
> > > > name.
> > > >
> > > > For connector='metadata', it seems unclear to me (what is metadata of
> > > Flink
> > > > :D), so I don't recommend this option.
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > >
> > > > On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <
> > > gabor.g.somogyi@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for your comments, please find my answers inline.
> > > > >
> > > > > BR,
> > > > > G
> > > > >
> > > > >
> > > > > On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <zakelly.lan@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Gabor,
> > > > > >
> > > > > > Thanks for the FLIP! It's very convenient for users to use SQL to
> > > read
> > > > > > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > > > > > thoughts/questions:
> > > > > >
> > > > > >    1. I see this connector is designed to read the
> > > checkpoint/savepoint
> > > > > >    data, so I'd suggest naming this connector as 'checkpoint' or
> > > > > > 'savepoint',
> > > > > >    which is more intuitive. The "connector='state'" makes me
> think
> > > this
> > > > > is
> > > > > >    reading the runtime state. Also the 'state.path' can be
> changed
> > to
> > > > > >    'checkpoint.path'.
> > > > > >
> > > > > Rename is a possibility, I'm fine with either way in terms of
> naming.
> > > > > The only thing what I would consider is that later on it would be
> > good
> > > > > to add metadata reading possibility. Such case we can either
> provide
> > > > > "savepoint" for data reading and "savepoint_metadata" or simply
> > > > "metadata"
> > > > > for meta reading (See 3rd bullet for another approach). Please
> share
> > > your
> > > > > thoughts on this.
> > > > >
> > > > >
> > > > > >    2. IIUC this connector is a bounded source and batch execution
> > is
> > > > > >    preferred, right? And it cannot read checkpoints while the
> > > original
> > > > > job
> > > > > > is
> > > > > >    running and doing periodic checkpoints. Thus I'd prefer the
> name
> > > > > >    'savepoint', as the lifecycle of savepoints is managed by the
> > user
> > > > > > instead
> > > > > >    of the Flink job.
> > > > > >
> > > > > We can call it savepoint, just agree on the metadata naming as well
> > to
> > > > have
> > > > > an overall picture.
> > > > >
> > > > >
> > > > > >    3. As you pictured, there might be more connectors for states
> > > other
> > > > > than
> > > > > >    keyed state, so I'm thinking about providing them in one
> > connector
> > > > > with
> > > > > > an
> > > > > >    option 'state.type' enumerating 'keyed',  'non-keyed'
> > > ('operator'),
> > > > > and
> > > > > > so
> > > > > >    on. Well for the first version this is not a required option.
> > > Seems
> > > > > more
> > > > > >    organized. WDYT?
> > > > > >
> > > > > Yeah, later on we must provide such config parameter somehow.
> > > > > Couple of things to consider:
> > > > > - Maybe we can add "metadata" as possible value here instead of
> > > > > "connector=X"
> > > > > - I'm thinking about whether we can come up with something
> > > > > where we can mix metadata, keyed and non-keyed data in a single SQL
> > > > > statement.
> > > > > Without super deep consideration I would say no, but I'm going to
> > think
> > > > it
> > > > > through.
> > > > >
> > > > >
> > > > > >    4. I'm not sure if the general option 'key.format' is needed,
> > > would
> > > > > >    using 'fields.k.format' while the 'k' is the name of the SQL
> > > primary
> > > > > > column
> > > > > >    be feasible?
> > > > > >
> > > > > I think that's a nice catch of a simplification.
> > > > >
> > > > >
> > > > > >    5. Reading window state[1] is unsupported, right?
> > > > > >
> > > > > Yeah, not yet covered here and can come in a later FLIP.
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <
> > > > gabor.g.somogyi@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for
> > keyed
> > > > > state
> > > > > > > data [1].
> > > > > > > Feel free to add your thoughts to make this feature better.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > > > > >
> > > > > > > BR,
> > > > > > > G
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Zakelly,

Just for my own clear understanding the suggestion is the following?

state.type=keyed|non-keyed|operator|metadata

Where non-keyed == operator?
I'm fine with this.

BR,
G


On Thu, Dec 19, 2024 at 10:31 AM Zakelly Lan <za...@gmail.com> wrote:

> Hi Gabor,
>
> The proposed configs look good to me. I also suggest that 'operator' can
> serve as an equivalent to 'non-keyed', as non-keyed state is also called
> operator state [1].
>
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#operator-state
>
> Best,
> Zakelly
>
> On Thu, Dec 19, 2024 at 4:46 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
> > Hi Zakelly,
> >
> > Fair point to have short names.
> > How about having the following configs then?
> >
> > connector=savepoint
> > state.type=keyed|non-keyed|metadata
> >
> > This would introduce the proper keyed/non-keyed option + the config pair
> > will
> > exactly tell that savepoint metadata is going to be read.
> >
> > BR,
> > G
> >
> >
> > On Thu, Dec 19, 2024 at 4:11 AM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi Gabor,
> > >
> > > Thanks for your answers!
> > >
> > > I'm good with naming it 'savepoint' with an extra option like 'mode'
> > > specifying the reading type, or suffixes in names like
> > 'savepoint-metadata'
> > > for each type of state. For the second option it means the current
> > > connector will be named 'savepoint-keyed'? Well I slightly prefer a
> short
> > > name.
> > >
> > > For connector='metadata', it seems unclear to me (what is metadata of
> > Flink
> > > :D), so I don't recommend this option.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > >
> > > On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <
> > gabor.g.somogyi@gmail.com>
> > > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thanks for your comments, please find my answers inline.
> > > >
> > > > BR,
> > > > G
> > > >
> > > >
> > > > On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Gabor,
> > > > >
> > > > > Thanks for the FLIP! It's very convenient for users to use SQL to
> > read
> > > > > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > > > > thoughts/questions:
> > > > >
> > > > >    1. I see this connector is designed to read the
> > checkpoint/savepoint
> > > > >    data, so I'd suggest naming this connector as 'checkpoint' or
> > > > > 'savepoint',
> > > > >    which is more intuitive. The "connector='state'" makes me think
> > this
> > > > is
> > > > >    reading the runtime state. Also the 'state.path' can be changed
> to
> > > > >    'checkpoint.path'.
> > > > >
> > > > Rename is a possibility, I'm fine with either way in terms of naming.
> > > > The only thing what I would consider is that later on it would be
> good
> > > > to add metadata reading possibility. Such case we can either provide
> > > > "savepoint" for data reading and "savepoint_metadata" or simply
> > > "metadata"
> > > > for meta reading (See 3rd bullet for another approach). Please share
> > your
> > > > thoughts on this.
> > > >
> > > >
> > > > >    2. IIUC this connector is a bounded source and batch execution
> is
> > > > >    preferred, right? And it cannot read checkpoints while the
> > original
> > > > job
> > > > > is
> > > > >    running and doing periodic checkpoints. Thus I'd prefer the name
> > > > >    'savepoint', as the lifecycle of savepoints is managed by the
> user
> > > > > instead
> > > > >    of the Flink job.
> > > > >
> > > > We can call it savepoint, just agree on the metadata naming as well
> to
> > > have
> > > > an overall picture.
> > > >
> > > >
> > > > >    3. As you pictured, there might be more connectors for states
> > other
> > > > than
> > > > >    keyed state, so I'm thinking about providing them in one
> connector
> > > > with
> > > > > an
> > > > >    option 'state.type' enumerating 'keyed',  'non-keyed'
> > ('operator'),
> > > > and
> > > > > so
> > > > >    on. Well for the first version this is not a required option.
> > Seems
> > > > more
> > > > >    organized. WDYT?
> > > > >
> > > > Yeah, later on we must provide such config parameter somehow.
> > > > Couple of things to consider:
> > > > - Maybe we can add "metadata" as possible value here instead of
> > > > "connector=X"
> > > > - I'm thinking about whether we can come up with something
> > > > where we can mix metadata, keyed and non-keyed data in a single SQL
> > > > statement.
> > > > Without super deep consideration I would say no, but I'm going to
> think
> > > it
> > > > through.
> > > >
> > > >
> > > > >    4. I'm not sure if the general option 'key.format' is needed,
> > would
> > > > >    using 'fields.k.format' while the 'k' is the name of the SQL
> > primary
> > > > > column
> > > > >    be feasible?
> > > > >
> > > > I think that's a nice catch of a simplification.
> > > >
> > > >
> > > > >    5. Reading window state[1] is unsupported, right?
> > > > >
> > > > Yeah, not yet covered here and can come in a later FLIP.
> > > >
> > > >
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <
> > > gabor.g.somogyi@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I'd like to start a discussion of FLIP-496: SQL connector for
> keyed
> > > > state
> > > > > > data [1].
> > > > > > Feel free to add your thoughts to make this feature better.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > > > >
> > > > > > BR,
> > > > > > G
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Zakelly Lan <za...@gmail.com>.
Hi Gabor,

The proposed configs look good to me. I also suggest that 'operator' can
serve as an equivalent to 'non-keyed', as non-keyed state is also called
operator state [1].


[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#operator-state

Best,
Zakelly

On Thu, Dec 19, 2024 at 4:46 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi Zakelly,
>
> Fair point to have short names.
> How about having the following configs then?
>
> connector=savepoint
> state.type=keyed|non-keyed|metadata
>
> This would introduce the proper keyed/non-keyed option + the config pair
> will
> exactly tell that savepoint metadata is going to be read.
>
> BR,
> G
>
>
> On Thu, Dec 19, 2024 at 4:11 AM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Gabor,
> >
> > Thanks for your answers!
> >
> > I'm good with naming it 'savepoint' with an extra option like 'mode'
> > specifying the reading type, or suffixes in names like
> 'savepoint-metadata'
> > for each type of state. For the second option it means the current
> > connector will be named 'savepoint-keyed'? Well I slightly prefer a short
> > name.
> >
> > For connector='metadata', it seems unclear to me (what is metadata of
> Flink
> > :D), so I don't recommend this option.
> >
> >
> > Best,
> > Zakelly
> >
> >
> > On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <
> gabor.g.somogyi@gmail.com>
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for your comments, please find my answers inline.
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi Gabor,
> > > >
> > > > Thanks for the FLIP! It's very convenient for users to use SQL to
> read
> > > > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > > > thoughts/questions:
> > > >
> > > >    1. I see this connector is designed to read the
> checkpoint/savepoint
> > > >    data, so I'd suggest naming this connector as 'checkpoint' or
> > > > 'savepoint',
> > > >    which is more intuitive. The "connector='state'" makes me think
> this
> > > is
> > > >    reading the runtime state. Also the 'state.path' can be changed to
> > > >    'checkpoint.path'.
> > > >
> > > Rename is a possibility, I'm fine with either way in terms of naming.
> > > The only thing what I would consider is that later on it would be good
> > > to add metadata reading possibility. Such case we can either provide
> > > "savepoint" for data reading and "savepoint_metadata" or simply
> > "metadata"
> > > for meta reading (See 3rd bullet for another approach). Please share
> your
> > > thoughts on this.
> > >
> > >
> > > >    2. IIUC this connector is a bounded source and batch execution is
> > > >    preferred, right? And it cannot read checkpoints while the
> original
> > > job
> > > > is
> > > >    running and doing periodic checkpoints. Thus I'd prefer the name
> > > >    'savepoint', as the lifecycle of savepoints is managed by the user
> > > > instead
> > > >    of the Flink job.
> > > >
> > > We can call it savepoint, just agree on the metadata naming as well to
> > have
> > > an overall picture.
> > >
> > >
> > > >    3. As you pictured, there might be more connectors for states
> other
> > > than
> > > >    keyed state, so I'm thinking about providing them in one connector
> > > with
> > > > an
> > > >    option 'state.type' enumerating 'keyed',  'non-keyed'
> ('operator'),
> > > and
> > > > so
> > > >    on. Well for the first version this is not a required option.
> Seems
> > > more
> > > >    organized. WDYT?
> > > >
> > > Yeah, later on we must provide such config parameter somehow.
> > > Couple of things to consider:
> > > - Maybe we can add "metadata" as possible value here instead of
> > > "connector=X"
> > > - I'm thinking about whether we can come up with something
> > > where we can mix metadata, keyed and non-keyed data in a single SQL
> > > statement.
> > > Without super deep consideration I would say no, but I'm going to think
> > it
> > > through.
> > >
> > >
> > > >    4. I'm not sure if the general option 'key.format' is needed,
> would
> > > >    using 'fields.k.format' while the 'k' is the name of the SQL
> primary
> > > > column
> > > >    be feasible?
> > > >
> > > I think that's a nice catch of a simplification.
> > >
> > >
> > > >    5. Reading window state[1] is unsupported, right?
> > > >
> > > Yeah, not yet covered here and can come in a later FLIP.
> > >
> > >
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <
> > gabor.g.somogyi@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> > > state
> > > > > data [1].
> > > > > Feel free to add your thoughts to make this feature better.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > > >
> > > > > BR,
> > > > > G
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Zakelly,

Fair point to have short names.
How about having the following configs then?

connector=savepoint
state.type=keyed|non-keyed|metadata

This would introduce the proper keyed/non-keyed option + the config pair
will
exactly tell that savepoint metadata is going to be read.

BR,
G


On Thu, Dec 19, 2024 at 4:11 AM Zakelly Lan <za...@gmail.com> wrote:

> Hi Gabor,
>
> Thanks for your answers!
>
> I'm good with naming it 'savepoint' with an extra option like 'mode'
> specifying the reading type, or suffixes in names like 'savepoint-metadata'
> for each type of state. For the second option it means the current
> connector will be named 'savepoint-keyed'? Well I slightly prefer a short
> name.
>
> For connector='metadata', it seems unclear to me (what is metadata of Flink
> :D), so I don't recommend this option.
>
>
> Best,
> Zakelly
>
>
> On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for your comments, please find my answers inline.
> >
> > BR,
> > G
> >
> >
> > On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi Gabor,
> > >
> > > Thanks for the FLIP! It's very convenient for users to use SQL to read
> > > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > > thoughts/questions:
> > >
> > >    1. I see this connector is designed to read the checkpoint/savepoint
> > >    data, so I'd suggest naming this connector as 'checkpoint' or
> > > 'savepoint',
> > >    which is more intuitive. The "connector='state'" makes me think this
> > is
> > >    reading the runtime state. Also the 'state.path' can be changed to
> > >    'checkpoint.path'.
> > >
> > Rename is a possibility, I'm fine with either way in terms of naming.
> > The only thing what I would consider is that later on it would be good
> > to add metadata reading possibility. Such case we can either provide
> > "savepoint" for data reading and "savepoint_metadata" or simply
> "metadata"
> > for meta reading (See 3rd bullet for another approach). Please share your
> > thoughts on this.
> >
> >
> > >    2. IIUC this connector is a bounded source and batch execution is
> > >    preferred, right? And it cannot read checkpoints while the original
> > job
> > > is
> > >    running and doing periodic checkpoints. Thus I'd prefer the name
> > >    'savepoint', as the lifecycle of savepoints is managed by the user
> > > instead
> > >    of the Flink job.
> > >
> > We can call it savepoint, just agree on the metadata naming as well to
> have
> > an overall picture.
> >
> >
> > >    3. As you pictured, there might be more connectors for states other
> > than
> > >    keyed state, so I'm thinking about providing them in one connector
> > with
> > > an
> > >    option 'state.type' enumerating 'keyed',  'non-keyed' ('operator'),
> > and
> > > so
> > >    on. Well for the first version this is not a required option. Seems
> > more
> > >    organized. WDYT?
> > >
> > Yeah, later on we must provide such config parameter somehow.
> > Couple of things to consider:
> > - Maybe we can add "metadata" as possible value here instead of
> > "connector=X"
> > - I'm thinking about whether we can come up with something
> > where we can mix metadata, keyed and non-keyed data in a single SQL
> > statement.
> > Without super deep consideration I would say no, but I'm going to think
> it
> > through.
> >
> >
> > >    4. I'm not sure if the general option 'key.format' is needed, would
> > >    using 'fields.k.format' while the 'k' is the name of the SQL primary
> > > column
> > >    be feasible?
> > >
> > I think that's a nice catch of a simplification.
> >
> >
> > >    5. Reading window state[1] is unsupported, right?
> > >
> > Yeah, not yet covered here and can come in a later FLIP.
> >
> >
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <
> gabor.g.somogyi@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> > state
> > > > data [1].
> > > > Feel free to add your thoughts to make this feature better.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > >
> > > > BR,
> > > > G
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Zakelly Lan <za...@gmail.com>.
Hi Gabor,

Thanks for your answers!

I'm good with naming it 'savepoint' with an extra option like 'mode'
specifying the reading type, or suffixes in names like 'savepoint-metadata'
for each type of state. For the second option it means the current
connector will be named 'savepoint-keyed'? Well I slightly prefer a short
name.

For connector='metadata', it seems unclear to me (what is metadata of Flink
:D), so I don't recommend this option.


Best,
Zakelly


On Wed, Dec 18, 2024 at 10:35 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi Zakelly,
>
> Thanks for your comments, please find my answers inline.
>
> BR,
> G
>
>
> On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Gabor,
> >
> > Thanks for the FLIP! It's very convenient for users to use SQL to read
> > checkpoint data, so overall I'd +1 for this FLIP. Here are my
> > thoughts/questions:
> >
> >    1. I see this connector is designed to read the checkpoint/savepoint
> >    data, so I'd suggest naming this connector as 'checkpoint' or
> > 'savepoint',
> >    which is more intuitive. The "connector='state'" makes me think this
> is
> >    reading the runtime state. Also the 'state.path' can be changed to
> >    'checkpoint.path'.
> >
> Rename is a possibility, I'm fine with either way in terms of naming.
> The only thing what I would consider is that later on it would be good
> to add metadata reading possibility. Such case we can either provide
> "savepoint" for data reading and "savepoint_metadata" or simply "metadata"
> for meta reading (See 3rd bullet for another approach). Please share your
> thoughts on this.
>
>
> >    2. IIUC this connector is a bounded source and batch execution is
> >    preferred, right? And it cannot read checkpoints while the original
> job
> > is
> >    running and doing periodic checkpoints. Thus I'd prefer the name
> >    'savepoint', as the lifecycle of savepoints is managed by the user
> > instead
> >    of the Flink job.
> >
> We can call it savepoint, just agree on the metadata naming as well to have
> an overall picture.
>
>
> >    3. As you pictured, there might be more connectors for states other
> than
> >    keyed state, so I'm thinking about providing them in one connector
> with
> > an
> >    option 'state.type' enumerating 'keyed',  'non-keyed' ('operator'),
> and
> > so
> >    on. Well for the first version this is not a required option. Seems
> more
> >    organized. WDYT?
> >
> Yeah, later on we must provide such config parameter somehow.
> Couple of things to consider:
> - Maybe we can add "metadata" as possible value here instead of
> "connector=X"
> - I'm thinking about whether we can come up with something
> where we can mix metadata, keyed and non-keyed data in a single SQL
> statement.
> Without super deep consideration I would say no, but I'm going to think it
> through.
>
>
> >    4. I'm not sure if the general option 'key.format' is needed, would
> >    using 'fields.k.format' while the 'k' is the name of the SQL primary
> > column
> >    be feasible?
> >
> I think that's a nice catch of a simplification.
>
>
> >    5. Reading window state[1] is unsupported, right?
> >
> Yeah, not yet covered here and can come in a later FLIP.
>
>
> >
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
> >
> > Best,
> > Zakelly
> >
> > On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <gabor.g.somogyi@gmail.com
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> state
> > > data [1].
> > > Feel free to add your thoughts to make this feature better.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > >
> > > BR,
> > > G
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Zakelly,

Thanks for your comments, please find my answers inline.

BR,
G


On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi Gabor,
>
> Thanks for the FLIP! It's very convenient for users to use SQL to read
> checkpoint data, so overall I'd +1 for this FLIP. Here are my
> thoughts/questions:
>
>    1. I see this connector is designed to read the checkpoint/savepoint
>    data, so I'd suggest naming this connector as 'checkpoint' or
> 'savepoint',
>    which is more intuitive. The "connector='state'" makes me think this is
>    reading the runtime state. Also the 'state.path' can be changed to
>    'checkpoint.path'.
>
Rename is a possibility, I'm fine with either way in terms of naming.
The only thing what I would consider is that later on it would be good
to add metadata reading possibility. Such case we can either provide
"savepoint" for data reading and "savepoint_metadata" or simply "metadata"
for meta reading (See 3rd bullet for another approach). Please share your
thoughts on this.


>    2. IIUC this connector is a bounded source and batch execution is
>    preferred, right? And it cannot read checkpoints while the original job
> is
>    running and doing periodic checkpoints. Thus I'd prefer the name
>    'savepoint', as the lifecycle of savepoints is managed by the user
> instead
>    of the Flink job.
>
We can call it savepoint, just agree on the metadata naming as well to have
an overall picture.


>    3. As you pictured, there might be more connectors for states other than
>    keyed state, so I'm thinking about providing them in one connector with
> an
>    option 'state.type' enumerating 'keyed',  'non-keyed' ('operator'), and
> so
>    on. Well for the first version this is not a required option. Seems more
>    organized. WDYT?
>
Yeah, later on we must provide such config parameter somehow.
Couple of things to consider:
- Maybe we can add "metadata" as possible value here instead of
"connector=X"
- I'm thinking about whether we can come up with something
where we can mix metadata, keyed and non-keyed data in a single SQL
statement.
Without super deep consideration I would say no, but I'm going to think it
through.


>    4. I'm not sure if the general option 'key.format' is needed, would
>    using 'fields.k.format' while the 'k' is the name of the SQL primary
> column
>    be feasible?
>
I think that's a nice catch of a simplification.


>    5. Reading window state[1] is unsupported, right?
>
Yeah, not yet covered here and can come in a later FLIP.


>
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state
>
> Best,
> Zakelly
>
> On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I'd like to start a discussion of FLIP-496: SQL connector for keyed state
> > data [1].
> > Feel free to add your thoughts to make this feature better.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> >
> > BR,
> > G
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Zakelly Lan <za...@gmail.com>.
Hi Gabor,

Thanks for the FLIP! It's very convenient for users to use SQL to read
checkpoint data, so overall I'd +1 for this FLIP. Here are my
thoughts/questions:

   1. I see this connector is designed to read the checkpoint/savepoint
   data, so I'd suggest naming this connector as 'checkpoint' or 'savepoint',
   which is more intuitive. The "connector='state'" makes me think this is
   reading the runtime state. Also the 'state.path' can be changed to
   'checkpoint.path'.
   2. IIUC this connector is a bounded source and batch execution is
   preferred, right? And it cannot read checkpoints while the original job is
   running and doing periodic checkpoints. Thus I'd prefer the name
   'savepoint', as the lifecycle of savepoints is managed by the user instead
   of the Flink job.
   3. As you pictured, there might be more connectors for states other than
   keyed state, so I'm thinking about providing them in one connector with an
   option 'state.type' enumerating 'keyed',  'non-keyed' ('operator'), and so
   on. Well for the first version this is not a required option. Seems more
   organized. WDYT?
   4. I'm not sure if the general option 'key.format' is needed, would
   using 'fields.k.format' while the 'k' is the name of the SQL primary column
   be feasible?
   5. Reading window state[1] is unsupported, right?


[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/#window-state

Best,
Zakelly

On Wed, Dec 18, 2024 at 5:46 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi All,
>
> I'd like to start a discussion of FLIP-496: SQL connector for keyed state
> data [1].
> Feel free to add your thoughts to make this feature better.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
>
> BR,
> G
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
If I understand correctly you use purely SQL workloads for savepoint
creation. In short such case the initial version won't be much help.

In a more detailed version SQL operators are creating non-keyed/operator
state data which is going to be supported later on.
The first version supports user code defined keyed state data.

BR,
G


On Tue, Dec 24, 2024 at 4:45 AM Venkatakrishnan Sowrirajan <vs...@asu.edu>
wrote:

> Hi Gabor,
>
> - Expose state metadata on data stream API
> > - Add metadata support for the SQL connector
> > - Add non-keyed/operator state data support for the SQL connector
> > - Add state catalog
>
>
> Thanks for this FLIP. This will be a great addition. Although, most of our
> use cases are SQL and not DataStream which seems to only benefit when
> "non-keyed/operator
> state data support for the SQL connector" is added. Is my understanding
> correct?
>
> Regards
> Venkata krishnan
>
>
> On Mon, Dec 23, 2024 at 6:59 PM Shengkai Fang <fs...@gmail.com> wrote:
>
> > Thanks for your clarification. Looking forward to the following FLIPs.
> >
> > Best,
> > Shengkai
> >
> > Gabor Somogyi <ga...@gmail.com> 于2024年12月23日周一 18:32写道:
> >
> > > Hi Shengkai,
> > >
> > > Thanks for your comments.
> > >
> > > Both state catalog and metadata tables are on the roadmap but not in
> this
> > > FLIP.
> > > The main approach is to do things incrementally in order to avoid
> > > infinite discussion threads and 10k+ lines pull requests.
> > >
> > > Just to give some teaser, this is how I can roughly imagine upcoming
> > FLIPs
> > > (unordered):
> > > - Expose state metadata on data stream API
> > > - Add metadata support for the SQL connector
> > > - Add non-keyed/operator state data support for the SQL connector
> > > - Add state catalog
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fs...@gmail.com>
> wrote:
> > >
> > > > Hi.
> > > >
> > > > Thanks Gabor's great FLIP. It is very useful for many cases. But I
> have
> > > > some suggestions about this FLIP:
> > > >
> > > > 1. It's better to introduce a state Catalog that helps users to
> > translate
> > > > the state information to a table schema. The state catalog uses
> > > > `<operator_name>`.`<state_name>` to find required state and uses
> state
> > > > descriptor to restore the schema.
> > > > 2. Can the connector provide more metadata information for users? For
> > > > example, the `ttl` field is very useful in our cases.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > >
> > > >
> > > >
> > > > Gabor Somogyi <ga...@gmail.com> 于2024年12月19日周四 21:41写道:
> > > >
> > > > > Hi Yanquan,
> > > > >
> > > > > Thanks for the question, please see the FLIP content related this.
> > > > >
> > > > > ---BEGIN FLIP CONTENT---
> > > > > The target of this implementation proposal is to provide SQL
> support
> > > for
> > > > > keyed states. Other types are going to be covered in further FLIP
> > > > > documents.
> > > > > ---END FLIP CONTENT---
> > > > >
> > > > > Keyed state data which is defined by users in for example process
> > > > > functions, operator or non-keyed state data is defined by internal
> > > Flink
> > > > > operators for example sources/sinks.
> > > > > In short, we're intended to add this support later on which means
> > that
> > > > > further configuration parameters needs to be added.
> > > > >
> > > > > BR,
> > > > > G
> > > > >
> > > > >
> > > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi, Gabor. Thanks for driving this FLIP and this immediately
> > reminded
> > > > me
> > > > > > of exploring the status of the Source connector like Kafka or
> > > MySQLCDC
> > > > to
> > > > > > understand the progress of reading information.
> > > > > >
> > > > > > In SourceOperator, the state of reader in contained in a
> ListState
> > > > named
> > > > > > `SourceReaderState`[1], however, the state type of
> > > `SourceReaderState`
> > > > is
> > > > > > `byte[]`, so I am worried that we can't get any understandable
> > > > > information
> > > > > > using this connector. Do we consider supporting this scenario?
> > > > > >
> > > > > > If we want to support this scenario, perhaps we need to add more
> > > > > > parameters like fields.#.state-version-serializer and
> > > > > > fields.#.state-version to To specify the
> org.apache.flink.core.io
> > > > > .SimpleVersionedSerializer[2]
> > > > > > and version to be used, And it also requires more complex
> > > > implementation.
> > > > > >
> > > > > > Do I have any misunderstandings? Is this requirement something we
> > > need
> > > > to
> > > > > > support, or will it need support in the future.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java*L109__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6KTlxdJI$
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java*L50__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6giBFR_Y$
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com>
> 写道:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for
> > keyed
> > > > > state
> > > > > > > data [1].
> > > > > > > Feel free to add your thoughts to make this feature better.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-496*3A*SQL*connector*for*keyed*state*data__;JSsrKysrKw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6980yUcw$
> > > > > > >
> > > > > > > BR,
> > > > > > > G
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Hi Gabor,

- Expose state metadata on data stream API
> - Add metadata support for the SQL connector
> - Add non-keyed/operator state data support for the SQL connector
> - Add state catalog


Thanks for this FLIP. This will be a great addition. Although, most of our
use cases are SQL and not DataStream which seems to only benefit when
"non-keyed/operator
state data support for the SQL connector" is added. Is my understanding
correct?

Regards
Venkata krishnan


On Mon, Dec 23, 2024 at 6:59 PM Shengkai Fang <fs...@gmail.com> wrote:

> Thanks for your clarification. Looking forward to the following FLIPs.
>
> Best,
> Shengkai
>
> Gabor Somogyi <ga...@gmail.com> 于2024年12月23日周一 18:32写道:
>
> > Hi Shengkai,
> >
> > Thanks for your comments.
> >
> > Both state catalog and metadata tables are on the roadmap but not in this
> > FLIP.
> > The main approach is to do things incrementally in order to avoid
> > infinite discussion threads and 10k+ lines pull requests.
> >
> > Just to give some teaser, this is how I can roughly imagine upcoming
> FLIPs
> > (unordered):
> > - Expose state metadata on data stream API
> > - Add metadata support for the SQL connector
> > - Add non-keyed/operator state data support for the SQL connector
> > - Add state catalog
> >
> > BR,
> > G
> >
> >
> > On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fs...@gmail.com> wrote:
> >
> > > Hi.
> > >
> > > Thanks Gabor's great FLIP. It is very useful for many cases. But I have
> > > some suggestions about this FLIP:
> > >
> > > 1. It's better to introduce a state Catalog that helps users to
> translate
> > > the state information to a table schema. The state catalog uses
> > > `<operator_name>`.`<state_name>` to find required state and uses state
> > > descriptor to restore the schema.
> > > 2. Can the connector provide more metadata information for users? For
> > > example, the `ttl` field is very useful in our cases.
> > >
> > > Best,
> > > Shengkai
> > >
> > >
> > >
> > >
> > > Gabor Somogyi <ga...@gmail.com> 于2024年12月19日周四 21:41写道:
> > >
> > > > Hi Yanquan,
> > > >
> > > > Thanks for the question, please see the FLIP content related this.
> > > >
> > > > ---BEGIN FLIP CONTENT---
> > > > The target of this implementation proposal is to provide SQL support
> > for
> > > > keyed states. Other types are going to be covered in further FLIP
> > > > documents.
> > > > ---END FLIP CONTENT---
> > > >
> > > > Keyed state data which is defined by users in for example process
> > > > functions, operator or non-keyed state data is defined by internal
> > Flink
> > > > operators for example sources/sinks.
> > > > In short, we're intended to add this support later on which means
> that
> > > > further configuration parameters needs to be added.
> > > >
> > > > BR,
> > > > G
> > > >
> > > >
> > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com>
> > wrote:
> > > >
> > > > > Hi, Gabor. Thanks for driving this FLIP and this immediately
> reminded
> > > me
> > > > > of exploring the status of the Source connector like Kafka or
> > MySQLCDC
> > > to
> > > > > understand the progress of reading information.
> > > > >
> > > > > In SourceOperator, the state of reader in contained in a ListState
> > > named
> > > > > `SourceReaderState`[1], however, the state type of
> > `SourceReaderState`
> > > is
> > > > > `byte[]`, so I am worried that we can't get any understandable
> > > > information
> > > > > using this connector. Do we consider supporting this scenario?
> > > > >
> > > > > If we want to support this scenario, perhaps we need to add more
> > > > > parameters like fields.#.state-version-serializer and
> > > > > fields.#.state-version to To specify the org.apache.flink.core.io
> > > > .SimpleVersionedSerializer[2]
> > > > > and version to be used, And it also requires more complex
> > > implementation.
> > > > >
> > > > > Do I have any misunderstandings? Is this requirement something we
> > need
> > > to
> > > > > support, or will it need support in the future.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java*L109__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6KTlxdJI$
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java*L50__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6giBFR_Y$
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I'd like to start a discussion of FLIP-496: SQL connector for
> keyed
> > > > state
> > > > > > data [1].
> > > > > > Feel free to add your thoughts to make this feature better.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-496*3A*SQL*connector*for*keyed*state*data__;JSsrKysrKw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6980yUcw$
> > > > > >
> > > > > > BR,
> > > > > > G
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Shengkai Fang <fs...@gmail.com>.
Thanks for your clarification. Looking forward to the following FLIPs.

Best,
Shengkai

Gabor Somogyi <ga...@gmail.com> 于2024年12月23日周一 18:32写道:

> Hi Shengkai,
>
> Thanks for your comments.
>
> Both state catalog and metadata tables are on the roadmap but not in this
> FLIP.
> The main approach is to do things incrementally in order to avoid
> infinite discussion threads and 10k+ lines pull requests.
>
> Just to give some teaser, this is how I can roughly imagine upcoming FLIPs
> (unordered):
> - Expose state metadata on data stream API
> - Add metadata support for the SQL connector
> - Add non-keyed/operator state data support for the SQL connector
> - Add state catalog
>
> BR,
> G
>
>
> On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fs...@gmail.com> wrote:
>
> > Hi.
> >
> > Thanks Gabor's great FLIP. It is very useful for many cases. But I have
> > some suggestions about this FLIP:
> >
> > 1. It's better to introduce a state Catalog that helps users to translate
> > the state information to a table schema. The state catalog uses
> > `<operator_name>`.`<state_name>` to find required state and uses state
> > descriptor to restore the schema.
> > 2. Can the connector provide more metadata information for users? For
> > example, the `ttl` field is very useful in our cases.
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> > Gabor Somogyi <ga...@gmail.com> 于2024年12月19日周四 21:41写道:
> >
> > > Hi Yanquan,
> > >
> > > Thanks for the question, please see the FLIP content related this.
> > >
> > > ---BEGIN FLIP CONTENT---
> > > The target of this implementation proposal is to provide SQL support
> for
> > > keyed states. Other types are going to be covered in further FLIP
> > > documents.
> > > ---END FLIP CONTENT---
> > >
> > > Keyed state data which is defined by users in for example process
> > > functions, operator or non-keyed state data is defined by internal
> Flink
> > > operators for example sources/sinks.
> > > In short, we're intended to add this support later on which means that
> > > further configuration parameters needs to be added.
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com>
> wrote:
> > >
> > > > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded
> > me
> > > > of exploring the status of the Source connector like Kafka or
> MySQLCDC
> > to
> > > > understand the progress of reading information.
> > > >
> > > > In SourceOperator, the state of reader in contained in a ListState
> > named
> > > > `SourceReaderState`[1], however, the state type of
> `SourceReaderState`
> > is
> > > > `byte[]`, so I am worried that we can't get any understandable
> > > information
> > > > using this connector. Do we consider supporting this scenario?
> > > >
> > > > If we want to support this scenario, perhaps we need to add more
> > > > parameters like fields.#.state-version-serializer and
> > > > fields.#.state-version to To specify the org.apache.flink.core.io
> > > .SimpleVersionedSerializer[2]
> > > > and version to be used, And it also requires more complex
> > implementation.
> > > >
> > > > Do I have any misunderstandings? Is this requirement something we
> need
> > to
> > > > support, or will it need support in the future.
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> > > state
> > > > > data [1].
> > > > > Feel free to add your thoughts to make this feature better.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > > >
> > > > > BR,
> > > > > G
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Shengkai,

Thanks for your comments.

Both state catalog and metadata tables are on the roadmap but not in this
FLIP.
The main approach is to do things incrementally in order to avoid
infinite discussion threads and 10k+ lines pull requests.

Just to give some teaser, this is how I can roughly imagine upcoming FLIPs
(unordered):
- Expose state metadata on data stream API
- Add metadata support for the SQL connector
- Add non-keyed/operator state data support for the SQL connector
- Add state catalog

BR,
G


On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fs...@gmail.com> wrote:

> Hi.
>
> Thanks Gabor's great FLIP. It is very useful for many cases. But I have
> some suggestions about this FLIP:
>
> 1. It's better to introduce a state Catalog that helps users to translate
> the state information to a table schema. The state catalog uses
> `<operator_name>`.`<state_name>` to find required state and uses state
> descriptor to restore the schema.
> 2. Can the connector provide more metadata information for users? For
> example, the `ttl` field is very useful in our cases.
>
> Best,
> Shengkai
>
>
>
>
> Gabor Somogyi <ga...@gmail.com> 于2024年12月19日周四 21:41写道:
>
> > Hi Yanquan,
> >
> > Thanks for the question, please see the FLIP content related this.
> >
> > ---BEGIN FLIP CONTENT---
> > The target of this implementation proposal is to provide SQL support for
> > keyed states. Other types are going to be covered in further FLIP
> > documents.
> > ---END FLIP CONTENT---
> >
> > Keyed state data which is defined by users in for example process
> > functions, operator or non-keyed state data is defined by internal Flink
> > operators for example sources/sinks.
> > In short, we're intended to add this support later on which means that
> > further configuration parameters needs to be added.
> >
> > BR,
> > G
> >
> >
> > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com> wrote:
> >
> > > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded
> me
> > > of exploring the status of the Source connector like Kafka or MySQLCDC
> to
> > > understand the progress of reading information.
> > >
> > > In SourceOperator, the state of reader in contained in a ListState
> named
> > > `SourceReaderState`[1], however, the state type of `SourceReaderState`
> is
> > > `byte[]`, so I am worried that we can't get any understandable
> > information
> > > using this connector. Do we consider supporting this scenario?
> > >
> > > If we want to support this scenario, perhaps we need to add more
> > > parameters like fields.#.state-version-serializer and
> > > fields.#.state-version to To specify the org.apache.flink.core.io
> > .SimpleVersionedSerializer[2]
> > > and version to be used, And it also requires more complex
> implementation.
> > >
> > > Do I have any misunderstandings? Is this requirement something we need
> to
> > > support, or will it need support in the future.
> > >
> > >
> > > [1]
> > >
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109
> > > [2]
> > >
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50
> > >
> > >
> > >
> > >
> > >
> > > > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> > > >
> > > > Hi All,
> > > >
> > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> > state
> > > > data [1].
> > > > Feel free to add your thoughts to make this feature better.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > > >
> > > > BR,
> > > > G
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Shengkai Fang <fs...@gmail.com>.
Hi.

Thanks Gabor's great FLIP. It is very useful for many cases. But I have
some suggestions about this FLIP:

1. It's better to introduce a state Catalog that helps users to translate
the state information to a table schema. The state catalog uses
`<operator_name>`.`<state_name>` to find required state and uses state
descriptor to restore the schema.
2. Can the connector provide more metadata information for users? For
example, the `ttl` field is very useful in our cases.

Best,
Shengkai




Gabor Somogyi <ga...@gmail.com> 于2024年12月19日周四 21:41写道:

> Hi Yanquan,
>
> Thanks for the question, please see the FLIP content related this.
>
> ---BEGIN FLIP CONTENT---
> The target of this implementation proposal is to provide SQL support for
> keyed states. Other types are going to be covered in further FLIP
> documents.
> ---END FLIP CONTENT---
>
> Keyed state data which is defined by users in for example process
> functions, operator or non-keyed state data is defined by internal Flink
> operators for example sources/sinks.
> In short, we're intended to add this support later on which means that
> further configuration parameters needs to be added.
>
> BR,
> G
>
>
> On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com> wrote:
>
> > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded me
> > of exploring the status of the Source connector like Kafka or MySQLCDC to
> > understand the progress of reading information.
> >
> > In SourceOperator, the state of reader in contained in a ListState named
> > `SourceReaderState`[1], however, the state type of `SourceReaderState` is
> > `byte[]`, so I am worried that we can't get any understandable
> information
> > using this connector. Do we consider supporting this scenario?
> >
> > If we want to support this scenario, perhaps we need to add more
> > parameters like fields.#.state-version-serializer and
> > fields.#.state-version to To specify the org.apache.flink.core.io
> .SimpleVersionedSerializer[2]
> > and version to be used, And it also requires more complex implementation.
> >
> > Do I have any misunderstandings? Is this requirement something we need to
> > support, or will it need support in the future.
> >
> >
> > [1]
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109
> > [2]
> >
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50
> >
> >
> >
> >
> >
> > > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> > >
> > > Hi All,
> > >
> > > I'd like to start a discussion of FLIP-496: SQL connector for keyed
> state
> > > data [1].
> > > Feel free to add your thoughts to make this feature better.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> > >
> > > BR,
> > > G
> >
> >
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Yanquan,

Thanks for the question, please see the FLIP content related this.

---BEGIN FLIP CONTENT---
The target of this implementation proposal is to provide SQL support for
keyed states. Other types are going to be covered in further FLIP documents.
---END FLIP CONTENT---

Keyed state data which is defined by users in for example process
functions, operator or non-keyed state data is defined by internal Flink
operators for example sources/sinks.
In short, we're intended to add this support later on which means that
further configuration parameters needs to be added.

BR,
G


On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <de...@gmail.com> wrote:

> Hi, Gabor. Thanks for driving this FLIP and this immediately reminded me
> of exploring the status of the Source connector like Kafka or MySQLCDC to
> understand the progress of reading information.
>
> In SourceOperator, the state of reader in contained in a ListState named
> `SourceReaderState`[1], however, the state type of `SourceReaderState` is
> `byte[]`, so I am worried that we can't get any understandable information
> using this connector. Do we consider supporting this scenario?
>
> If we want to support this scenario, perhaps we need to add more
> parameters like fields.#.state-version-serializer and
> fields.#.state-version to To specify the org.apache.flink.core.io.SimpleVersionedSerializer[2]
> and version to be used, And it also requires more complex implementation.
>
> Do I have any misunderstandings? Is this requirement something we need to
> support, or will it need support in the future.
>
>
> [1]
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109
> [2]
> https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50
>
>
>
>
>
> > 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> >
> > Hi All,
> >
> > I'd like to start a discussion of FLIP-496: SQL connector for keyed state
> > data [1].
> > Feel free to add your thoughts to make this feature better.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> >
> > BR,
> > G
>
>

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Posted by Yanquan Lv <de...@gmail.com>.
Hi, Gabor. Thanks for driving this FLIP and this immediately reminded me of exploring the status of the Source connector like Kafka or MySQLCDC to understand the progress of reading information.

In SourceOperator, the state of reader in contained in a ListState named `SourceReaderState`[1], however, the state type of `SourceReaderState` is `byte[]`, so I am worried that we can't get any understandable information using this connector. Do we consider supporting this scenario?

If we want to support this scenario, perhaps we need to add more parameters like fields.#.state-version-serializer and fields.#.state-version to To specify the org.apache.flink.core.io.SimpleVersionedSerializer[2] and version to be used, And it also requires more complex implementation.

Do I have any misunderstandings? Is this requirement something we need to support, or will it need support in the future.


[1] https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109
[2] https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50





> 2024年12月18日 17:43,Gabor Somogyi <ga...@gmail.com> 写道:
> 
> Hi All,
> 
> I'd like to start a discussion of FLIP-496: SQL connector for keyed state
> data [1].
> Feel free to add your thoughts to make this feature better.
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data
> 
> BR,
> G